One of the difficulties with implementing a software factory along with best takeoff software is measuring development-team performance. Depending on the culture of the team, and the desired outcomes for your team, the approach may be fundamentally different, but here is an approach I have implemented, that I believe is fairly unique in a corporate environment.
My Objectives were:
- To change the culture from one that was inward-looking (every problem is solved by writing some code) to one that considered external commercial or Open Source solutions (a problem can be solved in many different ways).
- To reward outcomes rather than inputs, i.e. the primary concern is that the project objectives, the deadlines and the architectural principles and standards were all met.
- To allow developers some freedom to choose their projects
- To encourage information sharing, collaboration and cross-skilling within the team
- To reward the top performers and manage the poor performers
This looks like a lot to ask from a single process with a simple measurement system, but I believe that the process we have selected should achieve all of these. Firstly it is necessary to note that I have created three development teams who are accountable for progressing our technical strategy and defining standards and toolsets within their respective domains. These are the “Front-Ends” team which includes the web-team, the Integration team, and the Services team.
The RFP Process
- At the Project prioritisation forum the Business-Unit heads agree the priorities of the projects for which we will publish RFPs
- The Project office prepares a very brief RFP describing the key outcomes required for the project.
- As the CIO, I estimate the size of the project and award the project a size-value on a scale of 1-20.
- The RFP is published on an internal blog indicating its size-value and priority.
- The developers are notified of a new RFP and may then respond to the RFP or form a syndicate across the various teams to respond to the RFP. Only two days are allocated for the construction of a proposal at this point in the process.
- The responses are considered at the weekly Architecture council meeting (Chief Architect, Head of Operations, Heads of each development team, Data & Services registrar) in terms of the quality of the technical approach, the reasonableness of the effort estimate and the time-to-market vs architectural-significance trade-offs.
- One syndicate is awarded the project and must take into consideration any guidance offered by the architectural council.
The Planning Process
- The syndicate should then embark on a detailed estimation process which follows a Delphi methodology to arrive at a fairly good estimate of effort.
- A detailed design process is worked into the plan and after the design is complete the effort estimates are reviewed one more time.
- The project is baselined at this point and deadline targets are measured based on this iteration of the project estimation.
- Depending on each syndicate members involvement after the estimation the project-size units earned by each member is adjusted if necessary.
The Project Scoring process
- Once the project has been completed it must be scored by all the relevant stakeholders.
- A standardized electronic survey with a set of questions for each type of stakeholder is used, so that scores can be easily compared.
- The standardized questions are published so that everyone knows in advance what the required outcomes are.
The Performance Measurement Metrics:
Using these measures the following can easily be measured and managed:
- Each individual must have earned at least 20 Project size points in a year. I measure this on a prorated basis every quarter and meet with underperforming team members to help them get back on track.
- In order to qualify for a bonus each individual must earn at least 24 project-size points in a year.
- Of those eligible for a bonus, their individual Aggregate Project Outcome Scores (Sum of (earned Projects Size-points x Project Survey Score) for all the projects an individual was involved in) determines their rank on the Bonus list. Top 5 scorers earn 15,14,13,12,11% of the development team bonus pool respectively, and the balance of the eligible team-members share the remaining 37% as a ratio of their relative scores.
Performance Reporting
Although all this may sound complicated it reduces each team-member’s score to two metrics: Project Size Points and Project Outcome Scores.
- If you have more than 20 Project Size points you keep your job.
- If you have more than 24 Project Size points you will get a bonus.
- The size of that bonus is determined by you Project outcome Scores (The quality & timeliness of your work)
The current scores for each team member (both size and outcome scores) are published weekly for all to see. No-one should have any doubt where they stand at any point in the year.
Some Balancing factors
Of course none of this works if there are insufficient projects to keep everyone busy during the year. It would not be fair if there were not enough project-size points available to ensure that everyone can earn their required 20. Fortunately I do not anticipate that scenario for many years to come.
I am sure that by the time that occurs humans will no longer be involved in software development processes, and performance measurement will have a completely different set of objectives.
Add to Technorati Favorites
Subscribe to Sober Counsel
Thanks Graeme.
Love the idea of mentorship points. Could do something like, if the syndicate leader brings someone on to the syndicate who is not knowledgable in that domain, and that person can demonstrate they have gained this knowledge during the project then the project is awarded an extra point. Will think about how to apply the idea more carefully.
There is definitely resistance from some who say I am bribing them with the bonus; Interestingly those very people have the lowest delivery rate. As a recruitment attraction it is working very well: The best people who know they can score well are very keen to work here.
thanks again, will definitely post an update at the end of the financial year when the ratings are done.
Malcolm,
Great innovations here. Please keep us updated on the outcomes and progress. My interest is always in the human side of these systems, and how people are motivated (or not). My only concern would be with how the prioritisation system of allocating projects would work. If someone consistently felt that they were not being given their first choice projects, and were being forced to do projects they didn’t like (to gain annual project points), this would be no better than a normal approach of simply telling people what to do. The manager’s role in the system you propose is vital, specifically in helping those who might be brilliant at delivery but not so good at promoting themselves or writing proposals.
It might also be good to award points for mentorship in this process, so that the best and brightest don’t continually work together at the exclusion of new or weaker members of the team.
Best of luck.
Thanks Paul.
I deeply value the insight you bring. Of course the measurement of the other areas of IT are also a concern, but one of the most difficult has always been measuring development performance, because it is both a creative and deeply technical area. This mechanism is quite specifically applied to the development team members, Even the team leads have a slightly different set of criteria.
I will certainly try to work in peer review process and stress the creative element of the proposal evaluation step. In fact in some scenario’s it has already come up, where two syndicates presented on one project, and both answered the brief, but the most innovative won. It drew a lot of “correspondence” from the syndicate who was not awarded the project.
thank you again, and I am encouraged by your comment to publish more about my approach across the range of IT concerns.
Malcolm, you describe a thought provoking approach.
From my own experience, I have found there are many facets to leading IT in an organisation, and the production of software is just one of them. It would seem to me that while a software factory you describe might suit some developers style, this view may not be universal. Your success will likely hinge on the accuracy and percieved fairness of the points allocated per project and consequently per person.
If I may presume, productivity of developers is very much influenced by their pride in their works, and how their peers rate their effectiveness. Might I suggest you add the concept of peer review, with a certain number of your points allocated during the process rather than just at the start… This may serve as a motivator to those for whom the art is more important than money.
I second the sentiment of another commentator who said that creativity/innovation in software does not normally stem from outside of IT, as people not specialising in the field lack the context to do so. Some provision should be made for internal projects.
Kind regards
Thanks Paul.
This is very constructive and useful comment. When we pioneer these sorts of things it is always useful to have an array of opinions to temper it with.
I hope it is quite progressive, and seems to be a big attraction for new and younger hires.
Malcolm,
This is a good article in the theory/practice of human performance engineering. In my experience entropy rules, so with this in mind any system will
a) gravitate to chaos
b) follow the path of least resistance.
In your case you have some good measures in place to control the “chaos”, however you may have to keep tabs on the “resistance” our system imposes. If your model is not functioning as the least resistance path, an alrernative will develop.
I really like the syndication / developer pool model. This is by far the most free-enterprising model I have seen in a corporate setting (reminds me a bit of http://www.freelancer.com)
Further to this, you may have to look into distinguising those projects who build business enablers v.s. those who build business solutions. The enablers are far more valuable in terms of architecural building blocks, but often far less exciting or visible to the business. Your points system may have to take this into account, or balance the required points between foundational work and front-end business visible work.
Lastly, innovation / great ideas seldom come out of commissioned work, so don’t expect this level of action out of the RFP part of your process. If you are looking for innovative approaches and ideas, you might want to look into adding eiter;
a) room for innovation through direct initiatives
b) competition.
I prefer b) since humans are inertly competitive. In your model awarding the same project to two competing internal syndicates should bring out the right level of creative action in a corporate setting, especially if you add measurements that drive alternative thinking.
Thanks for the comment Dylan. Yup. I had the idea to add size points to a project as a sweetener for various reasons. So if it is a particularly urgent project, or a team agrees to take on a trainee etc I could add some size points to that project. We’ve been going with this process, about 5 months now, and seems to be working…. We’ll see how it goes.
Great article… Its nice to see how its all come together. 🙂 While this seems like a lot of red tape, meetings, pain and proposals… its a system that will work to benefit the company and those that work hard in it. The only problem I’d see, is if you get a new guy who has no connections to team up with and no extensive knowledge of the system to go at it alone. It may be beneficial to throw in some bonus points for sharing the knowledge, “Apprenticeship points” should keep the flow of knowledge through the team going strong!