People involved in software projects would say that software development is about understanding complexity.
What is complexity?
What can we do about it?
Complexity is easy to define at a high level, but people get vague when pressed for a precise definition. Let's see if we can:
What is complexity?
What can we do about it?
- quantify the problem
- define it
- suggest how to address it
Assume we need to develop a project of 50,000 lines of code 1
, this project would take 5,000 man days to develop (scenario A) assuming 10
production lines of code per day 2. Even if we are generous
and assume 50 production lines of code per day per developer, the project would
still take 1,000 man days to develop.
Now take that developed code and have a single developer re-enter it (scenario B). The developer is likely to be able to code at least 1,000 lines of code per day and it would only take about 50 man days to enter the entire program.
The difference between scenario A and scenario B is 950 days to create the same software system. So what are we doing in the other 950 days if it only takes 50 days to write the code?
95% of a project is about solving problems, not writing code
In scenario A we don't know which lines of code to write, time is expended in figuring out:
Projects do not fail because their is insufficient time to write code
When you see developers furiously coding at their desks, they are spending most of their time thinking.
Most developers are unable to sit and plan away from the computer, so when you see them at their keyboards they are primarily thinking. They will write and rewrite sections of code as they think through the problems (see Not Planning is for Losers).
1 Lines of code is a terrible metric and is only used here to illustrate a point about complexity. Software measurement should use function points or another relevant metric.
2 Brooks, Frederick
Phillips. “The Mythic Man Month”, 1995. ISBN 0-201-00650-2
Now take that developed code and have a single developer re-enter it (scenario B). The developer is likely to be able to code at least 1,000 lines of code per day and it would only take about 50 man days to enter the entire program.
The difference between scenario A and scenario B is 950 days to create the same software system. So what are we doing in the other 950 days if it only takes 50 days to write the code?
95% of a project is about solving problems, not writing code
In scenario A we don't know which lines of code to write, time is expended in figuring out:
- determining what the actual requirements are
- translating the requirements into high level design
- designing an architecture sufficient to support the requirements
- translating the high level design into programming languages / libraries
- discovering where the code defects are
- rewriting code for incorrect functionality
While it might seem startling that 95% of our time is spend figuring out which lines of code to write, a little reflection will convince us that this is the case. We mistakenly believe that projects fail because there is not enough time to write the code, but really, the problem is that we don't know what lines of code to write.
Most developers are unable to sit and plan away from the computer, so when you see them at their keyboards they are primarily thinking. They will write and rewrite sections of code as they think through the problems (see Not Planning is for Losers).
Where does Complexity Come From?
Now that we have quantified how much time is spent solving complexity, where does it come from?
Complexity comes from team resources not understanding how to solve the problems they face. When confronted with a problem the team must think about how to solve it.
Each problem is like figuring out a path in a maze, it takes time to solve it initially. Once you have solved the problem, it is easy to avoid all the dead ends and go straight to a solution.
We spend 95% of our time considering alternatives and hitting dead ends.
Complexity is not a issue if enough time has been budgeted in the project for the team to think things through. Complexity is a problem when you need to think through issues and insufficient time is budgeted.
Projects run late mainly because of two reasons:
Complexity comes from team resources not understanding how to solve the problems they face. When confronted with a problem the team must think about how to solve it.
We spend 95% of our time considering alternatives and hitting dead ends.
Complexity is not a issue if enough time has been budgeted in the project for the team to think things through. Complexity is a problem when you need to think through issues and insufficient time is budgeted.
Projects run late mainly because of two reasons:
- Insufficient time
- You did not use formal estimation techniques and underestimate the project
- You use formal estimates and executives push for a delivery date independent of the problems to be solved in the project
- Insufficient requirements
- You estimated sufficient time for your team's skill level to solve the problems but you did not have all the requirements
Insufficient time
It is pretty clear why insufficient time causes projects to fail, but here are some specific statistics. Often executives reject formal estimates and then push for project deadlines independent of the work to be done (see Why Senior Management Declared Deadlines lead to Disaster)
Rejection of estimates for business reasons lead to productivity down 15% and quality down 21%
Excessive schedule pressure leads to productivity down 16% and quality down 23%
Insufficient Requirements
The second situation happens when you underestimate requirement complexity. On a regular basis projects are launched where only a fraction of the requirements are gathered. Once the project starts there is a mad dash to capture the missing requirements, i.e scope creep (see Shift Happens).
Project plans are often based on the initial set of requirements and the project deadline is not adjusted when you discover that there are missing requirements.
Failure to estimate requirement changes to schedule lead to productivity down 16% and quality down 22%
Why We Underestimate Requirement Complexity
Human beings are wired to underestimate complexity. For example, suppose that a customer says that they want a car, we all know what a car is, correct? Executives often talk in such generalities when they don't understand why the estimates are high.
Look at the following cars:
Look at the following cars:
Clearly these cars are not the same, and the cost and time required to build them is not the same. Abstractly, we all understand what a car is and we can discuss this in general. But when we start drilling into the details, there are many questions to be answered, such as:
- How fast does the car need to go?
- How many people does it need to seat?
- What kind of fuel economy do you want?
- What safety features are required?
- How sophisticated do the controls need to be?
In the rush to get projects started, we skimp on the requirements process and many questions are not answered. But at some point the developers need that information so that they can solve their problems and the requirements need to be gathered. Quite often projects run out of time simply because they never budgeted time to gather enough requirements.
Skill and Complexity
There is a clear trade off between the time required to solve a complex problem and the skill level of the team. A problem is complex when you have never seen it before, but once you have solved it, then you always see the solution.
Just like the maze shown here, the first time might take a fair amount of time to figure out the solution. If you have to solve the problem again then you will go directly to the solution without dead ends.
When gathering requirements, the better the analyst understands the domain, the less likely they are to miss requirements or have inconsistent requirements. This eliminates the likelihood of inconsistent requirements and increases the probability that enough requirements are captured to start the project.
When developers have experience with the domain then they take much less time to break the requirements into high level designs. When they have experience with the language being used and the libraries they can translate that high level design into a low level design and code much quicker.
Solving Complexity
Solving Complexity
So successful software projects recognize that complexity amounts to the thinking time required for the team to figure out problems they have not solved before.
The more experienced a resource, the less time they need to think about the final solution. The less experienced resources need more time to think through problems, especially when they hit dead ends.
The more experienced a resource, the less time they need to think about the final solution. The less experienced resources need more time to think through problems, especially when they hit dead ends.
- For the business analysts and product managers
- This is about understanding the subject matter and making sure to gather enough consistent requirements
- For the developers
- This is about understanding the subject matter for high level design, and understanding the language and libraries for implementation
- For the project managers
- This is about understanding how capable the team is and making sure that sufficient time is budgeted for them to solve all problems
- For the IT executives
- This is about understanding that any given project team needs sufficient time to solve problems and not caving into executive pressure to arbitrarily shorten projects
References
- Jones, Capers.SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
1 Lines of code is a terrible metric and is only used here to illustrate a point about complexity. Software measurement should use function points or another relevant metric.
No comments:
Post a Comment