No Experience Required!

Did you know that we have never found a relationship between a developer's years of experience and code quality or productivity?

The original study that found huge variations in individual programming productivity was conducted in the late 1960s by Sackman, Erikson, and Grant (1968).  

This study has been repeated at least 8 times over 30 years and the results have not changed! (see below) 

Sackman et al studied professional programmers with an average of 7 years’ experience and found that:

• the ratio of initial coding time was about 20 to 1
• the ratio of debugging times over 25 to 1
• program execution speed about 10 to 1
• program size 5 to 1

They found no relationship between a programmer's number of years of experience and code quality or productivity.  That is there was NO correlation between experience and productivity (i.e. ability to produce code) and there was NO correlation between experience and quality (i.e. minimizing defects) .

Think about that for a minute...

That is the worst and the best programmers made distinct groups and each group had people of both low and high experience levels.  Whether training helps developers or not is not indicated by these findings, only that years of experience do not matter in the aggregate.  If training does matter then very few developers are getting trained.

Without considering legality, it is simpler to get rid of expensive poor performers and hire good performers with few years of experience!

This does not mean that training does not make a difference for some people.  What it means is that unproductive people tend to stay unproductive over their entire career.

Results Have Been Confirmed for 30 Years!

There were flaws in the study, however, after accounting for the flaws, their data still shows more than an order of magnitude difference between the best programmers and the worst, and that difference was not related to experience.

In years since the original study, the general finding that "There are order-of-magnitude differences among programmers" has been confirmed by many other studies of professional programmers (full references at the end of the article):

• Curtis 1981
• Mills 1983
• DeMarco and Lister 1985
• Curtis et al. 1986
• Card 1987
• Boehm and Papaccio 1988
• Valett and McGarry 1989
• Boehm et al 2000

Technology is More Sophisticated, Developers are not

You might  think that we know much more about software development today since 1968, after all today: 

• we have better computer languages
• we have more sophisticated technology
• we have better research on effective software patterns
• we have formal software degrees available in university

It turns out that all these things are true, but we still have order of magnitude differences among programmers and the difference is not related to years of experience.  

That means that there is some other x-factor that drives productive developers;  that x-factor is probably the ability to plan and make good decisions.

The bad news is that if you are unproductive then years of experience won't make you better

Developers face making decisions on how to structure their code every day.  There is always a choice when it comes to:

• laying out code pathways
• packaging functions into classes
• packaging classes into modules

Because developers face coding complex decisions, the best developers will plan their work and make good decisions.

Bad developers just 'jump in'; they assume that they can always rewrite code or make up for bad decisions later. Bad developers are not even aware that their decision processes are poor and that they can become much better by planning their work.

When have you gotten time to go back and fix suboptimal decisions?

Solution hinted at by Watts Humphrey

Watts Humphrey tried to get developers to understand the value of planning development, and making decisions in the Personal Software Process (PSP) for individuals and the Team Software Process (TSP) for teams. An analysis of over 18,000 projects demonstrates that¹:

PSP can raise productivity by 21% and quality by 31%

TSP can raise productivity by 21% and quality by 31%

This can have a profound effect on the way that we build and train our teams.  Rather than having large teams of mediocre developers, it makes much more sense to have smaller teams of highly productive developers and train them on how to plan and make good decisions.  

The findings suggest that the best way to rehabilitate a poor developer is to teach them how to make better decisions.

Conclusion

We inherently know this, we just don't do it.  If the senior management of organizations only knew about these papers, we could make sure that the productive people get paid what they are worth and the non-productive people could seek employment in some other field.  This would not only reduce the cost of building software but also increase the quality of the software that is produced.

Unfortunately, we are doomed to religious battles where people debate methodologies, languages, and technologies in the foreseeable future.  The way that most organizations develop code makes voodoo look like a science!

Eventually we'll put the 'science' back in Computer Science, I just don't know if it will be in my lifetime...

Related Articles

• Productive Developers are Smart and Lazy
• Are You are Surrounded by Idiots? Unfortunately, You Might be the Idiot...

Bibliography

Boehm, Barry W., and Philip N. Papaccio. 1988. "Understanding and Controlling Software Costs." IEEE Transactions on Software Engineering SE-14, no. 10 (October): 1462-77.

Boehm, Barry, et al, 2000. Software Cost Estimation with Cocomo II, Boston, Mass.: Addison Wesley, 2000.

Card, David N. 1987. "A Software Technology Evaluation Program." Information and Software Technology 29, no. 6 (July/August): 291-300.

Curtis, Bill. 1981. "Substantiating Programmer Variability." Proceedings of the IEEE 69, no. 7: 846.

Curtis, Bill, et al. 1986. "Software Psychology: The Need for an Interdisciplinary Program." Proceedings of the IEEE 74, no. 8: 1092-1106.

DeMarco, Tom, and Timothy Lister. 1985. "Programmer Performance and the Effects of the Workplace." Proceedings of the 8th International Conference on Software Engineering. Washington, D.C.: IEEE Computer Society Press, 268-72.

¹Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.

Mills, Harlan D. 1983. Software Productivity. Boston, Mass.: Little, Brown.

Valett, J., and F. E. McGarry. 1989. "A Summary of Software Measurement Experiences in the Software Engineering Laboratory." Journal of Systems and Software 9, no. 2 (February): 137-48.

Death March Calculus

In the previous post Stop It! No… really stop it. we talked about 5 worst practices and their impact on a software project.  However, all 5 worst practices will be present in a Death March.

Ed Yourdon, one of the pioneers in software engineering, wrote "Death March" to describe the common practice of setting up a software project which is not feasible.

That is, between the time, resources, and functionality (quality) desired for the project, there is no way to complete the project successfully.   Ed states that a death march is one in which one of the project parameters exceeds the norm by at least 50%.

A death march project is often known by the entire project team to be virtually impossible to succeed at.  To understand some of the motivation of why senior management creates death marches or why people participate in them you would need to read Ed's book.


In the previous article we showed this table:

Worst Practice	Productivity	Quality
Friction/antagonism among team members	-12.0%	-15.0%
Friction/antagonism among management	-13.5%	-18.5%
Inadequate communications with stakeholders	-13.5%	-18.5%
Layoffs/loss of key personnel	-15.7%	-21.7%
Excessive schedule pressure	-16.0%	-22.5%

Since senior management has asked for specific functionality that can not be produced by the current team in the given time frame, the first problem is the presence of excessive schedule pressure.  So right at the start of the project your productivity will be down 16% and quality will be down 22.5%.

With a compressed time schedule there will not be enough time to capture all the requirements from the stakeholders.  You will have to settle for partial requirements and hope to interpolate the missing requirements.  This will cause inadequate communications with stakeholders and drop the productivity 13.5% and the quality by 18.5%.  Unfortunately, these effects are cumulative with the excessive schedule pressure and you will now be down 29.5% productivity and 41% on quality.

There is no shortage of opinions in a software team; in the best teams they combine their opinions to make the best decisions.  When time is compressed we don't make good decisions in our teams and end up with friction at the management and team level. This will cause us to have friction/antagonism among management because they will have different opinions on how to execute the death march;  antagonism in management will lead to friction/antagonism among team members. These two issues will combine to drop productivity by 27% and quality by 37%.  The cumulative effect of all practices will have productivity down 64.5% and quality down by 78%.

When the project is obviously not going well, people will quit and/or be fired.  In particular, key personnel have a tendency to leave because they are the ones that have the easiest time finding new employment. If this happens then the final numbers will be productivity down 86.2% and quality down by 98.7%.

If you don't believe the numbers then you have never been on a death march!

The problem is that excessive schedule pressure will inevitably cause the other worst practices to come into existence.  Starting a project where excessive schedule pressure is present is virtually a guarantee that failure will occur.  In fact, the only death marches that have a chance at success are small teams with projects that are 3-6 months long.

Possible Way Out

The only way to solve the problem of excessive schedule pressure is to balance the three elements (time, resources, and functionality) of the project until you have a feasible project.  Since project teams are relatively fixed this means:

• reducing the functionality that needs to be produced
• moving back the project end date
• both reducing functionality and moving back the project end date

Experienced project managers on a death march will be aggressively reducing scope as soon as the project starts.  But this is no guarantee that they will get rid of enough functionality to bring the project into a feasible state.

If your organization has put you on a death march then the only possible solution is to find another job.  Hoping that somehow a successful project will result when you start off with excessive schedule pressure is futile, the mathematics shows that project failure is a virtual certainty.

You have more chances of winning big in the lottery and retiring than of completing a death march successfully.

Stop It! No… really stop it.

There are 5 worst practices in software development that if stopped immediately will:

Improve your productivity by a minimum of 12% and improve quality by a minimum of 15%

These practices are so common that people assume that they are normal -- they are not, they are silent killers wherever they are present.

We hear the term best practices enough to know that we all have different definitions for it. Simply put, a best practice is one that increases the chance your project will succeed.

How often do we talk about worst practices?

More importantly, what about those worst practices in your organization that you don't do anything about?

---

When it comes to a worst practice, just stop it.

---

If your company is practicing even one worst practice in the list below it will kill all your productivity and quality.

To make matters worse, some of the worst practices will trigger other worst practices to come into play¹.

The 5 Worst Practices

The worst practices and their effect on productivity and quality are as follows:

Worst Practice	Productivity	Quality
Friction/antagonism among team members	-12%	-15%
Friction/antagonism among management	-14%	-19%
Inadequate communications with stakeholders	-14%	-19%
Layoffs/loss of key personnel	-16%	-22%
Excessive schedule pressure	-16%	-23%

Excessive Schedule Pressure

Excessive schedule pressure is present whenever any of the following are practiced:

• Senior management declares the project deadline
• Formal estimation methods are used and ignored
• Informal (or dubious) estimation methods are used

Excessive schedule pressure causes the following to happen:

• Poor and incomplete requirements lead to extreme scope creep
• Architecture is an afterthought
• Code is written too quickly
• QA is reduced or skipped

This alone can create a Death March project and virtually guarantee project failure.

Effect of excessive schedule pressure is that productivity will be down 16% and quality will be down 22%

Not only is excessive schedule pressure one of the worst practices it tends to drive the other worst practices:

• Friction amongst managers
• Friction amongst team members
• Increases the chance that key people leave the organization

---

If your organization has a habit of imposing excessive schedule pressure -- leave!

---

Friction Between People

Software development is a team activity in which we transform our intangible thoughts into tangible working code. Team spirit and collaboration is not an option if you want to succeed.

You don't have to like everyone on your team and you don't have to agree with all their decisions.

 You must understand that the team is more important than any single individual and learn to work through your differences.

---

Teams only work well when they are hard on the problem, not each other!

---

Managers disagree because of different perspectives on resource allocation, objectives, and requirements. Not being able to come to a consensus will cave in projects and make all the managers look bad.

Managers win together and lose together.

Effect of management friction is that productivity will be down 14% and quality will be down 19%

Friction among team members happens because of different perspectives on requirements, design, and priority. Again, everyone wins together and loses together -- you can not win and have anyone else lose.

Effect of team friction is that productivity will be down 12% and quality will be down 15%

You are wrong if you think that friction can be allowed and projects will succeed.

---

Any form of friction between managers or the team is deadly.

---

Inadequate Stakeholder Communication

Inadequate stakeholder communication comes in several forms:

• Not getting enough information on business objectives
• Not developing software in a transparent manner

Poor business objectives from stakeholders will lead to bad requirements  If you are not transparent in how you are developing the project then you can expect excessive schedule pressure from senior management.

Effect of inadequate stakeholder communication is that productivity will be down 14% and quality will be down 19%

Loss of Key Personnel

To add insult to injury, any of the other four worst practices above will lead to either:

• Key personnel leaving your organization
• Key personnel being layed off

The loss of key personnel through turnover is not a worst practice but a derivative practice of other worst practices.

Badly managed organizations and projects will cause the most competent people to seek better jobs elsewhere.

In addition, projects in distress tend to cut key personnel because they are expensive. Laying off key personnel will sandbag your ability to get back on track; laying off non-performing people is the only way to advance a late project.

Effect of layoffs/loss of key personnel is that productivity will be down 16% and quality will be down 22%

---

Loss of key personnel has a dramatic effect  productivity and morale and directly affects product quality

---

Conclusion

Any of the worst practices mentioned above will cause a project to be late and deliver defective code.

If you are in an organization that habitually practices any of these worst practices then your only real option is to quit.

Worst practices tend to feed each other and cause a negative spiral such as:

• Excessive schedule pressure (leads to)
• Management and team friction (leads to)
• Loss of key personnel

---

Senior managers that allow any of these practices can count on canceled projects and/or highly defective products

---

^{See also:}

• Inspections are not Optional 
• Evidence that 1 hour of inspections reduces 4 hours of QA
• No Experience Required
• Evidence that years of experience does not improve productivity

^Bibliography

• Gilb, Tom and Graham, Dorothy; Software Inspections; Addison Wesley, Reading, MA; 1993; ISBN 10: 0201631814.
• ¹Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
• Radice, Ronald A.; High Qualitiy Low Cost Software Inspections; Paradoxicon Publishingl Andover, MA; ISBN 0-9645913-1-6; 2002; 479 pages.
• Wiegers, Karl E.; Peer Reviews in Software – A Practical Guide; Addison Wesley Longman, Boston, MA; ISBN 0-201-73485-0; 2002; 232 pages.  

Testing Departments are for Losers

I understand that this is a very strong statement and I'm definitely not trying to insult anyone; so apologies in advance.  I'm trying to challenge the belief that testing is mandatory and that there should be one testing resource for every two developers.

Quality development is about quality assurance and zero defects not about having testing departments. Developers are the ones who inject defects into the code and therefore they are the best line of defense to remove them.  The developer has the best information on what needs to be tested in his code at the time that he writes it.  The longer it takes for testing or a customer to discover a code defect the longer the developer will spend in a debugger chasing down the problem.

One testing resource for every two developers is not a good solution

Quality Assurance (QA) is about making a positive statement about product quality.  QA is about positive assurance, which is stating,  “We are certain that there are few, if any, defects in this product.”  Airplanes are a product with quality assured, the manufacturers will stand by their quality and the statistics back them up.  Contrast this with this article or this article which wonders what would happen if Microsoft made airplanes -- would you fly in them?  This is not to say that software needs to have the same quality level as an airplane, only that whatever quality level you aspire to should be assured.

The reality is that most testing departments simply discover defects and forward them back to the engineering department to fix.  By the time the software product gets released we are basically saying, “We got rid of as many defects as we could find before management forced us to release this product, however, we really have no idea how many other defects are in the code”.  This is not assuring quality; at best you get negative assurance out of this.

Everyone understand that buggy software kills sales (and start-ups :-)), however, testing is often an after thought in many organizations.  When software products take longer than expected, the testing department is often expected to test and bless code in less time than allocated.

To compound problems, many testing departments don't even receive proper requirements against which to test the code and/or sufficient tools to work with. Large testing departments and/or large amounts of manual testing are not healthy or efficient.

Humphrey Watts was emphatic that calling defects “bugs” trivializes the issue and downplays the negative impact that defects cause on a development organization.

Calling defects "bugs" trivializes an important issue

Defects are not introduced into software by goblins and elves.  Defects are injected into the code by developers that:

• don't understand the requirements or architecture
• misunderstand how to use their peer's components
• misunderstand 3rd party libraries
• having a bad day because of home troubles or work environment
• are careless because someone else will test their code

Defects are injected by the team

No one is more aware of how code can break down than the developer who writes it.   Any line of code that is written without concentration and planning becomes a potential defect.  It is impossible for testers to understand every pathway through the code and make sure that every possible combination of variables is properly taken care of.

There are many techniques that can increase code quality and dramatically reduce the amount of testing that is necessary:

• test driven development (TDD)
• database driven testing
• design by contract (DbC)
• inspections
• pair programming
• minimizing cyclomatic complexity
• using static and dynamic code tools
• proper code planning techniques

Test Driven Development

Properly written tests require a developer not only to think about what a code section is supposed to do but also plan how the code will be structured.  If you know that there are  five pathways through the code then you will write five tests ahead of time.  A common problem is that you have coded n paths through the code when there are n+1 conditions.

TDD is white box testing and can reach every pathway that the developer codes.  TDD is proactive and can test pathways from end to end, it does not just have to be used for unit testing.  When TDD is hooked up to a continuous integration engine then defects are located and fixed before they make it to testing.

Database Driven Testing

Using actual test data to test existing routines during development is an excellent way to make sure that there are fewer production problems.  The test data needs to be a copy (or subset) of production data.

Database driven testing can also be hooked up to a continuous integration engine and prevent defects from getting to testing.

Design By Contract

The Eiffel programming language introduced design by contract (DbC).  DbC is orthogonal to TDD because its goal is to ensure that the contract defined by the preconditions and postconditions for each function call is not violated.  DbC can be used in virtually any language for with their is an Aspect Oriented Programming (AOP) solution.

During development, the minute a developer violates the expected contract of any function (his or a peers) then the developer will get feedback to fix the problem before it gets to testing.

Inspections

Since the 1970s we have statistical evidence that one of the best ways to eliminate defects from code is through inspections.  Inspections can be applied to the requirements, design, and code artifacts and projects that use inspections can eliminate 99% of the defects injected into the code.  Se Inspections are not Optional and Software Professionals do Inspections.

Each hour of inspections will save you 4 hours of testing

Pair Programming

Pair programming can be selectively used to prevent and eliminate defects from code.   When developers work in pairs they not only review code as quickly as possible but also learn productivity techniques from each other.  Pair programming should only be done on complex sections of code. Pair programming not only eliminates defects but allows developers to get enough feedback that they can prevent defects in the future.

Minimizing Cyclomatic Complexity

There is evidence that routines with high cyclomatic complexity will have more latent defects than other routines.   This makes sense because the number of code pathways goes up dramatically as cyclomatic complexity increases and increases the chance that the developer does not handle all of them.   In most cases, testing departments can not reproduce all of the pathways in routines of high cyclomatic complexity.

Use Dynamic and Static Code Checking

There are many code problems caused by a careless use of pointers and other powerful language constructs.  Many of these problems can be detected by having the development team use dynamic and static code checking problems.

Proper Code Planning Techniques

There are developers that try to write code at the keyboard without planning, which is neither efficient nor effective.  This is like having to do errands in 5 locations and driving to the locations randomly -- you might get your errands done, but odds are it won't be efficient.

Watts Humphrey talks directly to the idea of planning in the Personal Software Process.  In addition techniques like diagramming with UML or using decision tables can go a long way to thinking through code structure before it is implemented.

Conclusion

Developers are the ones who inject defects into the code and therefore they are the best line of defense to remove them.  The developer has the best information on what needs to be tested in his code at the time that he writes it.  The longer it takes for testing or a customer to discover a code defect the longer the developer will spend in a debugger chasing down the problem.

Developers need to be trained in preventing and eliminating defects.  Developers who learn to get the code correct the first time will reduce and eliminate effort in testing.

One goal of development should be to catch defects early; this is the only way to assure quality.  Hence quality assurance starts and finishes in the development department, not the testing department.