What is software architecture?

What is software architecture? In the previous article "What is architecture?" we stated that architecture is about:

• Structural elements
• Connective elements

Let us talk about these with regard to a software system.  In our next article we'll talk about good and bad architecture.

Structural elements

Structural elements in software are layers that support other layers in the system:

• the language libraries are built on the functionality provided by the O/S
• the functionality of the software product is built on the language libraries
• 3rd part libraries that you are using are built on the previous 2 layers
• your code is built on all the previous layers

So any program you build has at least the following structural elements:

In a well designed system, your code will also break into layers specific to the application that you are building.  Each layer ideally only depends on layers underneath it, otherwise you will have circular references; asignal that there is a challenge or that something is wrong. 

Connective elements

Every software system has connective elements that communicate across the different layers.  Often 3rd party libraries are shared services that provide core services like data structures (XML, JSON, etc), logging, debugging, and other services.

Inside your code you will also have shared services that are used by every layer, this is diagrammed as follows:

Where layer 1, 2, ..., N represent the layers in your code and the share services are shown vertically as they cut across all the layers.  Layer N represents the UI layer of the system and the functionality at that layer depends on all the layers below it.

Generalizing Architecture

When people are talking about software architecture, they are not just talking about the structure of the code.  Often people are talking about the machines and O/S components as well.  So for example when people are talking about the LAMP architecture they are talking about:

• The O/S -- i.e. Linux
• The web server -- Apache, running on Linux
• The database -- MySql, running on Linux,
• The code -- PHP running on the MySql database using services from the O/S

Note: The PHP layer is further broken down like the diagram above in the Structural Elements section.

The LAMP architecture involves structural elements.  Connective elements are simply the ones created inside the PHP layer or generally provided by the O/S.

So when relevant, architecture includes:

• The O/S
• All support services, i.e. webserver, database, etc.
• The high level applications written by you or a 3rd party

General Software Architecture

For the structural elements of software we are taking about layers.  Those layers can be relative to the O/S such as how the O/S communicates with support services (web servers, databases, etc) or how your applications communicate with support services and the O/S.

Within your application there are different layer interactions, there is 1) how your base layers are built on the APIs of your language and the 3rd party libraries that you have installed, and 2) how you have broken up your layers to communicate with the different parts of your software.

The main structural elements of any model-view-controller (MVC) program looks somewhat like this:

• The model layer a.k.a. business layer is built on 3rd party APIs and O/S APIs
• The view layer a.k.a. GUI layer is built on 3rd party GUI APIs
• The data layer is build on database APIs

All of the APIs that you are using are based on the standard APIs of the language that you are using.

Implications

• Changing languages requires that every layer above it be reconsidered
• This means that if you change languages and some 3rd party API must be changed then the layer above also needs to be changed
• Strict separation in layers means that you can change a 3rd party API and only a single layer of the MVC system is affected
• Often developers color between the lines and a change of a 3rd party library causes quite a bit of pain as the layer infractions are found
• Not having enough of a business layer will be problematic because it is one of the main separating structures between your view/GUI and data layers.
• People discover lacking business layers when they change their database APIs or databases and have the problems propagate all the way through the GUI
• Often when business functionality is built into the model/GUI layer then that functionality is lost and needs to be recoded when a new GUI API is used

Connective elements

Connective software elements are used to shuttle data between the structural layers of the system. These connective elements are generally available everywhere (although they can be limited to a few layers).

Easily understood connective elements are those used in debugging and logging.  These elements are available universally, e.g. debugging and logging routines can be made available in all layers without worrying about layer dependency.

Other connective elements are most easily recognized because they are routines shared across layers of the architecture.  More specifically, they are the libraries/modules that are included into multiple layers of the program.

---

So now we have defined what structural and connective elements are in software, next we describe what it means to have good architecture and bad architecture.

Why Outsourcing Fails

Every year, we see corporations outsource operations only to pull back back some, if not all, of the operations. When this happens, the cost of pulling back operations or splitting things over multiple countries can leave you with significant operational issues and similar or worse costs.

Successful corporations understand the need to control costs in operations. The "spare no expense" philosophy of John Hammond in Jurassic Park is a one way ticket to disaster.

But, a focus on cost reduction is what leads corporations to face-fault when they outsource.

Corporations are seduced by the idea of potentially cutting costs dramatically; why pay expensive resources to do repetitive things when overseas people can do the same work for pennies?

The cardinal rule when outsourcing is:

Only operations that you completely understand and control can be outsourced 

To understand operations is to understand the costs of exceptions as well as how they are processed and escalated. Local resources deal efficiently with exceptions because they have informal relationships in the organization and use them to get things done efficiently; those relationships are not available to outsourced resources.

When operations gets outsourced, not only do foreign employees not understand how to deal with exceptions but also:

• They are often in another time zone 
• They don't have full access to local resources 
• They don't know who to contact inside the corporation 
• They don't understand local business rules 
• The network configuration can make it impossible to communicate properly 

Understanding an operational function means that you capture all details, including exceptions, in your information systems. Often, legacy information systems capture exception information as unstructured fields (i.e. memo fields) and off shore resources will not understand this.

99% of your operations might be straight forward, but the 1% of exceptions can at best lead to longer implementation times.  Best case scenario is that you will look bad to your customers and at worst, it will kill your bottom line.

If your information systems contain exception information in a structured way then at least you can eventually train foreign resources to resolve them. But unless exception information is captured so you can control it (i.e. not in general text fields) then it is the same thing as not having it.

When outsourced resources are under strict performance guidelines, they will do anything to clear their work queues. This often involves kicking back work they perceive as an exception as improperly specified; they will claim 'garbage in, garbage out'.

This will result in miscommunications and longer lead times to implement your services. Not only will delivery times, and errors go up, and your customers will get mad.

By the time you have fired the local employees, all the knowledge of the function has left the building and you are left with a broken system.

Key questions to get answers to:

• What does each exception in operations cost? 
• How often do they happen? 
• Can the information systems record exceptions in a way that you can track and control them? 
• Are the mechanisms that local resources use to resolve exceptions be used by outsourced resources? 

If you don't have a clear answer to each of these questions then outsourcing is likely to be a disaster. When you have a clear answer to the 3 questions and understand how you will address each exception then you will have much more success outsourcing.

Project failure is due to bad requirements

Projects can fail for a number of reasons, but at the root of most failures is a failure to gather correct and consistent requirements.  We've all laughed at some variant of the above diagram, but these issues are all because:

• We fail to capture correct and consistent requirements
• We play "broken telephone" when we are communicating requirements

Let's take a concrete example.  Suppose we have an orienteering challenge where you need to go from the start point below to the finish point, this is the actual requirement.  

But, there is a gap between what the actual requirements are and what is actually written down.

Good Requirements

As long as the written requirements don't diverge too much from the actual requirements, you will have time to adjust the requirements during project execution and still get to the finish point.  

So as long as the initial written requirements are in the green zone, you can still complete the project on time because the requirements point you in the correct direction.

Mediocre Requirements

Now suppose one of the following happen:

• You don't have all the core requirements because insufficient people were consulted
• You have conflicting requirements from different sources

When you have less than accurate requirements you will be in trouble.  The initial code and architecture of the project will be created based on the quality of the requirements.  If those requirements are suspect, then there will be rework as code needs to be adjusted as the scope of the project seems to shift.

So now you will find yourself in one of the yellow zones below

Even with the best execution, the project will be challenged.  Deadlines will be missed as you attempt to bring the requirements back to what they need to be.  This is like trying to change a tire on a car and discovering the jack is missing.

It is important to realize that adjusting poor requirements is not Scope Creep.  Fixing incorrect and inconsistent requirements is necessary and it is pointless for a project manager or executive to disallow these changes.  It would be worthwhile for project postmortems if the project manager tracked whether a requirement was missing or inconsistent.

Challenged projects are typically declared as successes, but only after massive compromises, burned out resources, damage to reputations, and loss of revenue.  When all the damage is taken into account, this is hardly a victory.

Bad Requirements

The last situation occurs where you only have vague requirements before you start a project.  This situation happens where the executives need a project done quickly and over-estimate the teams familiarity with the subject domain.

These projects start with requirements in the red zone below.  You don't have a prayer of completing this project and it will turn into a Death March with all of its characteristics (see Death March Calculus).

Making core course corrections to bad requirements are like trying to change a tire on a car when you are going down the highway at 100 mph.  It won't happen.

Conclusion

It is human nature to assume that the sooner you start a project that the sooner you will finish.   That assumption is only correct if you have good requirements to point you in the correct direction.  Good requirements are consistent and correct and include at a minimum the core requirements for the primary users.

Starting a project with mediocre or poor requirements is simply a recipe for project failure.  Mediocre or poor requirements are incomplete and inconsistent.

If you have been part of a project failure then you will discover that despite the other factors that went wrong, requirements failure was at the root of it.

Specifically, what is complexity?

People involved in software projects would say that software development is about understanding complexity.

What is complexity?

What can we do about it?

Complexity is easy to define at a high level, but people get vague when pressed for a precise definition.  Let's see if we can:

• quantify the problem
• define it
• suggest how to address it

Assume we need to develop a project of 50,000 lines of code ¹ , this project would take 5,000 man days to develop (scenario A) assuming 10 production lines of code per day ². Even if we are generous and assume 50 production lines of code per day per developer, the project would still take 1,000 man days to develop.

Now take that developed code and have a single developer re-enter it (scenario B). The developer is likely to be able to code at least 1,000 lines of code per day and it would only take about 50 man days to enter the entire program.

The difference between scenario A and scenario B is 950 days to create the same software system. So what are we doing in the other 950 days if it only takes 50 days to write the code?

95% of a project is about solving problems, not writing code

In scenario A we don't know which lines of code to write, time is expended in figuring out:

• determining what the actual requirements are
• translating the requirements into high level design
• designing an architecture sufficient to support the requirements
• translating the high level design into programming languages / libraries
• discovering where the code defects are
• rewriting code for incorrect functionality

While it might seem startling that 95% of our time is spend figuring out which lines of code to write, a little reflection will convince us that this is the case.  We mistakenly believe that projects fail because there is not enough time to write the code, but really, the problem is that we don't know what lines of code to write.

Projects do not fail because their is insufficient time to write code

When you see developers furiously coding at their desks, they are spending most of their time thinking.

Most developers are unable to sit and plan away from the computer, so when you see them at their keyboards they are primarily thinking. They will write and rewrite sections of code as they think through the problems (see Not Planning is for Losers).

Where does Complexity Come From?

Now that we have quantified how much time is spent solving complexity, where does it come from?

Complexity comes from team resources not understanding how to solve the problems they face. When confronted with a problem the team must think about how to solve it.

Each problem is like figuring out a path in a maze, it takes time to solve it initially.  Once you have solved the problem, it is easy to avoid all the dead ends and go straight to a solution.

We spend 95% of our time considering alternatives and hitting dead ends.

Complexity is not a issue if enough time has been budgeted in the project for the team to think things through. Complexity is a problem when you need to think through issues and insufficient time is budgeted.

Projects run late mainly because of two reasons:

1. Insufficient time
• You did not use formal estimation techniques and underestimate the project
• You use formal estimates and executives push for a delivery date independent of the problems to be solved in the project
2. Insufficient requirements
• You estimated sufficient time for your team's skill level to solve the problems but you did not have all the requirements

Insufficient time

It is pretty clear why insufficient time causes projects to fail, but here are some specific statistics.  Often executives reject formal estimates and then push for project deadlines independent of the work to be done (see Why Senior Management Declared Deadlines lead to Disaster)

Rejection of estimates for business reasons lead to productivity down 15% and quality down 21% 

Excessive schedule pressure leads to productivity down 16% and quality down 23% 

Insufficient Requirements

The second situation happens when you underestimate requirement complexity.  On a regular basis projects are launched where only a fraction of the requirements are gathered.  Once the project starts there is a mad dash to capture the missing requirements, i.e scope creep (see Shift Happens). 

Project plans are often based on the initial set of requirements and the project deadline is not adjusted when you discover that there are missing requirements.

Failure to estimate requirement changes to schedule lead to productivity down 16% and quality down 22% 

Why We Underestimate Requirement Complexity

Human beings are wired to underestimate complexity.  For example, suppose that a customer says that they want a car, we all know what a car is, correct?  Executives often talk in such generalities when they don't understand why the estimates are high.

Look at the following cars:

Clearly these cars are not the same, and the cost and time required to build them is not the same. Abstractly, we all understand what a car is and we can discuss this in general.  But when we start drilling into the details, there are many questions to be answered, such as:

• How fast does the car need to go?
• How many people does it need to seat?
• What kind of fuel economy do you want?
• What safety features are required?
• How sophisticated do the controls need to be?

In the rush to get projects started, we skimp on the requirements process and many questions are not answered.  But at some point the developers need that information so that they can solve their problems and the requirements need to be gathered.  Quite often projects run out of time simply because they never budgeted time to gather enough requirements.

Skill and Complexity

There is a clear trade off between the time required to solve a complex problem and the skill level of the team.  A problem is complex when you have never seen it before, but once you have solved it, then you always see the solution.

Just like the maze shown here, the first time might take a fair amount of time to figure out the solution.  If you have to solve the problem again then you will go directly to the solution without dead ends.

When gathering requirements, the better the analyst understands the domain, the less likely they are to miss requirements or have inconsistent requirements.  This eliminates the likelihood of inconsistent requirements and increases the probability that enough requirements are captured to start the project.  

When developers have experience with the domain then they take much less time to break the requirements into high level designs.  When they have experience with the language being used and the libraries they can translate that high level design into a low level design and code much quicker.


Solving Complexity

So successful software projects recognize that complexity amounts to the thinking time required for the team to figure out problems they have not solved before.

The more experienced a resource, the less time they need to think about the final solution.  The less experienced resources need more time to think through problems, especially when they hit dead ends.

• For the business analysts and product managers
• This is about understanding the subject matter and making sure to gather enough consistent requirements
• For the developers 
• This is about understanding the subject matter for high level design, and understanding the language and libraries for implementation
• For the project managers 
• This is about understanding how capable the team is and making sure that sufficient time is budgeted for them to solve all problems
• For the IT executives 
• This is about understanding that any given project team needs sufficient time to solve problems and not caving into executive pressure to arbitrarily shorten projects

References

• Jones, Capers.SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.

---

¹ Lines of code is a terrible metric and is only used here to illustrate a point about complexity. Software measurement should use function points or another relevant metric.

² Brooks, Frederick Phillips. “The Mythic Man Month”, 1995. ISBN 0-201-00650-2

Ready, Fire, Aim: How most gather requirements

Connecting with your customers and delivering value depends on understanding your customer's requirements and selling the correct product or solution that solves your customer's problems.

Commonly, the requirements gathering process is done hastily or not at all in the rush to get the sale. After all, the faster you can make the sales, the faster the money is in the bank -- correct?

The more that product customization is required, the more it is important understand the customer's actual problem.  Long lead times to build and implement a solution means will not only lead to difficult implementations but also to a more disappointed customer and a loss of future revenue.

If you build software, which has long lead times, it is even more important to make sure that what you are building will satisfy the customer's requirements as changing the final software solution will be nearly impossible and very costly.

Why do we continually misunderstand and sell the wrong solutions and building the wrong software?

Human Nature

Time pressures to make a sale put us under pressure and this stress leads to making quick decisions about whether a product or solution can be sold to a customer.  We listen to the customer but interpret everything he says according to the products and solutions that we have.

Often the customer will use words that seem to match exactly the products we have.  But then after the sale once we have to implement we often find that either we deliver a poor solution to the customer or a solution is infeasible and we must refund the customers money.

However, behind every word that the customer is using there is an implied usage, and understanding that implied usage is where we fail to gather requirements. For example, suppose the customer says I need a car.

Suppose that you sell used cars:

• the customer asked for a car
• you sell cars

Ergo problem solved!

• What if the customer needs an SUV but you don't have any?
• What if the customer really needs a truck?
• What if the customer needs a car with many modifications?
• What if the car the customer needs has never been built?

We hear the word car and we think that we know what the customer means.  The order-taker sales person will spring into action and sell what he thinks the customer needs. Behind the word car is an implied usage and unless you can ferret out the meaning that the customer has in mind, you are unlikely to sell the correct solution.

If you don't understand how the customer will to use your solution then you don't understand the problem.

Comedy of Errors

For products that require customization, the sale will get transferred to professional services that will dig deeper into the customer's requirements.  At this point you discover that the needs of the customer cannot be met.  This leads to sales people putting pressure on professional services and product management to find a solution, after all, losing the sale is not an option.

Sometimes heroic actions by the product management, professional services, and software development teams lead to a successful implementation, but usually not until there has been severe pain at the customer and midnight oil burned in your company.  You can eventually be successful but that customer will never buy from you again.

Consequences

As WIlliam Ralph Inge said, There are no rewards or punishments -- only consequences. The consequence of selling the wrong solution to a customer is:

• Whether you lose the sale or not, the customer loses faith in you
• The sales person is perceived as incompetent in the rest of the organization especially by professional services and software devleopment
• Your cost of sales and implementation is much higher than expected
• Your reputation is damaged

Conclusion

Selling the correct solution to the customer requires that you understand the customer's problem before you sell the solution.

When customization is required, good sales people engage resources that can capture the customer's requirements accurately and assess that you can deliver a solution to the customer.

Slowing down to understand the customer requirements and how you will solve his problems is the key.  By understanding the customer's requirements and producing the correct solutions you become a trusted adviser to the customer.

See also

• Shift Happens, effect of scope shift in a software development project.

Do Project Managers need Domain Experience?

Opinions vary on whether a project manager needs to have domain experience.  Certainly project managers that do not have domain experience will be the first to say that domain experience is not necessary as long as they have access to excellent subject matter experts.

I would advocate a more nuanced position; that is, a project manager does not need domain experience IF his subject matter experts understand the risks and dependencies that are inherent to the domain.

Let's go through a couple of personal projects that I have been involved with where the project manager did not have domain experience.

Telco Project

I am currently involved in a project that involves a LAN/WAN/WIFI upgrade of a large customer for a large telecommunications company.  The project manager does not have domain expertise in networks and is counting on the subject matter experts to provide him sufficient input to execute the project.

The subject matter experts are so advanced in their knowledge of networks that they no longer understand what beginners (i.e. the project manager) do not know.  They have assumed that when they indicate things to the project manager that he understands what they mean and will take appropriate actions.

The project manager is continually running into situations where he did not understand the implications of certain risks and dependencies.  This has caused a certain amount of rework and delays.

Fortunately, this is not a project with tremendous amounts of risk or dependencies so the project will be late but will succeed.

Mobile Handset Project

In the distant past, I was part of a team that was building a mobile POS terminal that worked over cellular (GSM, CDMA).  The project manager in this situation did not have domain experience and was counting on the subject matter experts.  In this case, the subject matter experts were very good at general design, but not experts in building cellular devices.

Because the subject matter experts were not specialists, they knew most of the key principles of designing mobile handsets but did not understand all the nuances of handset design.  There were several key issues required by practical handset manufacturing that were overlooked by the generalists and ended up creating such a strong cost over-run that the start-up went out of business.

Sumary

In the first project, the subject matter experts were extremely good, however, the project manager failed to understand the implication of some of their statements and this introduced large delays in the project.

In the second project, the subject matter experts were generalists and did not understand all the risks and dependencies of the project.  The project manager (and start-up) were doomed to fail because "you don't know what you don't know". Both these projects show that a project can be delayed or fail because a project manager does not have domain experience.

Conclusion

So if a project does not have many uncertainties and dependencies then it is extremely likely that the project manager does not require domain experience and can rely to some degree on his subject matter experts.

However, if the project has complex uncertainties and/or dependencies then a good project manager without domain experience is likely to find himself in a position where the consequences of not understanding the uncertainties and dependencies will either introduce serious rework or torpedo the project.

---

See also

• Schedule Risk is a Red Herring!!!
• Uncertainty trumps Risk in Software Development

Not using UML on Projects is Fatal

The Unified Modeling Language (UML) was adopted as a standard by the OMG in 1997, almost 20 years ago.  But despite its longevity, I'm continually surprised at few organizations actually use it.

Code is the ultimate model for software, but it is like the trees of a forest.  You can see a couple, but only few people can see the entire forest by just looking at the code.  For the rest of us, diagrams are the way to see the forest, and UML is the standard for diagrams.

They say, "A picture is worth a thousand words", and this is true for code; even on a large monitor you can only see so many lines of code.  Every other engineering discipline has diagrams for complex systems, e.g. design diagrams for airplanes, blueprints for buildings.  In fact, the diagrams need to be created and approved  BEFORE the airplane or building is created.

Contrast that with software where UML diagrams are rarely produced, or if they are produced, they are produced as an after thought.  The irony is that the people pushing to build the architecture quickly say that there is no time to make diagrams, but they are the first people to complain when the architecture sucks.  UML is key to planning (see Not planning is for losers)

I think this happens because developers, like all people, are focused on what they can see and touch right now.  It is easier to try to code a GUI interaction or tackle database update problems than it is to work at an abstract level through the interactions that are taking place from GUI to database

Yet this is where all the architecture is.  Good architecture makes all the difference in medium and large systems.  Architecture is the glue that holds the software components in place and defines communication through the structure.  If you don't plan the layers and modules of the system then you will continually be making compromises later on.

In particular, medium to large projects (>10,000 function points) are at a very high risk of failure if you don't consider the architectural issues.  Considering only 3 out of 10 software projects are successful only a fool would skip planning the architecture (see Failed? You get what you deserve!)

Good diagrams, in particular UML, allow you to abstract away all the low level details of an implementation and let you focus on planning the architecture.  This higher level planning leads to better architecture and therefore better extensibility and maintainability of software.

If you are a good coder then you will make a quantum leap in your ability to tackle large problems by being able to work through abstractions at a higher level.  How often do we find ourselves unable to implement simple features simply because the architecture doesn't support it?

Well the architecture doesn't support it because we spend very little time developing the blueprint for the architecture of the system.

UML diagrams need to be produced at two levels:

• the analysis or 'what' level
• the design or 'how' level

Analysis UML diagrams (class, sequence, collaboration) should be produced early in the project and support all the requirements.  Ideally you use a requirements methodology that allows you to trace easily from the requirements onto the diagrams.

Analysis diagrams do not have implementation classes on them, i.e. no vendor specific classes.  The goal is to identify how the high level concepts (user, warehouse, product, etc) relate to each other.

These analysis level UML diagrams will help you to identify gaps in the requirements before moving to design.  This way you can send your BAs and product managers back to collect missing requirements when you identify missing elements before you get too far down the road.

Once the analysis diagrams validate that the requirements are relatively complete and consistent, then you can create design diagrams with the implementation classes.  In general the analysis diagrams are one to many to the design diagrams.

Since you have validated the architecture at the analysis level, you can now do the design level without worrying about compromising the architectural integrity.  Once the design level is complete you can code without compromising the design level.

When well done the analysis UML, design UML, and code are all in sync.  Good software is properly planned and executed from the top down.  It is mentally tougher to create software this way, but the alternative is continuous patches and never ending bug-fix cycles.

So remember the following example from Covey's The 7 Principles of Highly Effective People:

You enter a clearing where a man is furiously sawing at a large log, but he is not making any progress.  You notice that the saw is dull and is unable to cut the wood, so you say, "Hey, if you sharpen the saw then you will saw the log faster".  To which the man replies, "I don't have time, I'm too busy sawing the log".

Don't be the guy sawing 
with a dull saw

UML is the tool to sharpen the saw, it does take time to learn and apply, but you will save yourself much more time and be much more successful.

Bibliography

• Covey, Stephen. The 7 Habits of Highly Effective People
• OMG, Unified Modeling Language™ (UML®) Resource Page

Infeasible Projects: Executive Ignorance or IT Impotence?

Infeasible software projects are launched all the time and teams are continually caught up in them, but what is the real source of the problem?

There are 2 year actual projects for which the executives set a 6 month deadline.

The project is guaranteed to fail but is this due to executive ignorance or IT impotence?

There is no schedule risk in an infeasible project because the deadline will be missed.  Schedule risk only exists in the presence of uncertainty (see Schedule Risk is a Red Herring!!!)

As you might expect, all executives and IT manager share responsibility for infeasible projects that turn into death marches.  Learn about the nasty side effects Death March Calculus. The primary causes for infeasible projects are:

• Rejection of formal estimates
• No estimation or improper estimation methods are used

Rejecting Formal Estimates

This situation occurs frequently; an example would be the Denver Baggage Handling System (see Case Study).

The project was automatically estimated (correctly) to take 2 years; however, executives declared that IT would only have 1 year to deliver.

Of course, they failed¹.

The deadline was rejected by executives because it did not fit their desires.  They could not have enjoyed the subsequent software disaster and bad press.

When executives ignore formal estimates they get what they deserve.  Formal estimates are ignored because executives believe through sheer force of will that they can set deadlines.

If IT managed to get the organization to pay for formal tools for estimating then it is not their problem that the executives refuse to go along with it.

Improper Estimation Methods

The next situation that occurs frequently is using estimation processes that have low validity.  Estimation has been extensively studied and documented by Tom DeMarco, Capers Jones, Ed Yourdon, and others.

Improper estimation methods will underestimate a software project every time. Fast estimates will be based on what you can think of, unfortunately, software is not tangible and so what you are aware of is like the tip of an iceberg.

None of this prevents executives demanding fast estimates from development.  Even worse, development managers will cave in to ridiculous demands and actually give fast estimates.

Poor estimates are guaranteed to lead to infeasible projects (see Who needs Formal Measurement?) Poor estimates are delivered by IT managers that:

• Can't convince executives to use formal tools
• Give in to extreme pressure for fast estimates

Infeasible projects that result from poor estimates are a matter of IT impotence.

Conclusion

Both executive ignorance and IT impotence lead to infeasible projects on a regular basis because of poor estimates and rejecting estimates; so there is no surprise here.

However, infeasible projects are a failure of executives and IT equally because we are all on the same team. It is not possible for part of the organization to succeed if the other one fails.

Possibly a greater share of problem is with IT management.  After all, whose responsibility is a bad decision -- the guys that know what the issues are or the ones that don't.

If a child wants ice cream before they eat dinner then whose fault is it if you cave in and give them the ice cream?

Unfortunately, even after 60 years of developing software projects, IT managers are either as ignorant as the executives or simply have no intestinal fortitude.

Even when IT managers convince executives of the importance of estimating tools, the estimates are routinely discarded because they do not meet executive expectations.

Rejection of automated estimates: productivity -16%, quality -22%

Until we can get a generation of IT managers that are prepared to educate executives on the necessity of proper estimation and be stubborn about holding to those estimates, we are likely to continue to have an estimated $3 trillion in failures of software projects every year.

---

End Notes

¹For inquiring minds, good automated estimation systems have been shown to be within 5% of time and cost on a regular basis. Contact me for additional information.

References

• Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
• Jones, Capers. The Economics of Software Quality. 2011
• Kahneman, Daniel. Thinking, Fast and Slow. 2011

Schedule Risk is a Red Herring!!!

We often hear the term schedule risk, however, it is generally a Red Herring. Stating that the schedule might stretch is about as useful as saying that eating can cause you to gain weight.

You may be correct but it gives you no leverage to solve the problem

Schedules slip as a result of a problem, if you want to solve the problem then you must identify the root cause.  Any problem will result in a task taking longer than expected and potentially affecting the schedule.

Risk and uncertainty are two sides of the same coin. Without uncertainty there is no risk.

No Uncertainty = No Risk

A risk is a contingent liability; a risk is a future event that is uncertain that has consequences. The key words are future and uncertain.

If 6 months remain and the deadline is in 2 months then there is no schedule risk because there is no uncertainty.

6 months late means that the earliest that the critical path items can finish is in 6 months. Just because the project has not hit the deadline and senior staff doesn't understand the project is late does not qualify the team to talk as if the outcome is uncertain.

It is disingenuous and cowardly to suggest to senior staff that a deadline is possible when you know that it is not.

When the team knows that they are late, they often talk about tasks as being risky simply because they hope that miracles can happen¹.

Hope is not a strategy

In fact, Kahneman points out all of us are wired to bet (pray?) on unlikely outcomes when faced with certain losses, i.e. we double down when faced with a loss.  Team members know about the negative consequences of failure and make projects seem possible simply because they want to delay the pain. Even worse, as the situation gets more desperate people will take bigger and bigger risks.

Using the term schedule risk when a project is not feasible essentially robs the managers of making a course correction until the point where very little can be done.

At a minimum, money can be saved by winding the project down. Few people have the intestinal fortitude to speak out when they know that a project is late. Unfortunately, cowardice is very common.

If you take a paycheck then you have an obligation to your organization to tell them when a project is late.

So it makes no sense to talk about schedule risk when:

• The project is late and you know it
• The project is not late but you see schedule items slipping

In the latter case you are much better to talk about why things are slipping rather than using the term schedule risk.  By talking about the root cause of the slippage, especially early in a project, can lead to you either solving the problem or adjusting the project deadline.  Either way, you will have a greater chance of ending up with a feasible project.

Related Articles

• Why Management Declared Deadlines Lead to Disaster
• No Business Case == Project Failure

References

• Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
• Jones, Capers. The Economics of Software Quality. 2011
• Kahneman, Daniel. Thinking, Fast and Slow. 2011

Failed? You get what you deserve!

Consider this, few projects fail because of unusual or unforeseen problems.  If you are trying to go so fast that you don't do your due diligence or decide to skip steps then -- "You get what you deserve!".

Projects succeed because they do not rely on force of personality and luck.

Only 3 of 10 software projects succeed (see Understanding Your Chances). Success happens when you implement best practices and avoid worst practices; not just talk about them and convince yourself that you are doing them.

Therefore 7 out of 10 projects fail, even if you redefine 'challenged projects' as political victories; these projects are often severely over time and budget.

It is estimated that between $3 trillion and $6 trillion dollars are wasted every year in IT; by organizations that "don't know, and don't know that they don't know", i.e. what you don't know can definitely hurt you.

What is amazing is that failures do not prompt the incompetent to learn why they failed.  After the post-fail finger-pointing ceremony, people just dust themselves off and rinse and repeat.

There is never enough time to do the project correctly, but there is always enough time to do it again.

Ingredients of a Successful Project

Successful software projects have the following characteristics:

• Effective capital budgeting
• Proper project sizing
• Estimate uncertainty
• A focus on defect removal

Capital Budgeting

Capital budgeting means having business cases for major projects and allocating capital to the most promising projects; a little due diligence prevents bad projects from starting.

For example, RJ Reynolds invested $325 million to develop smokeless cigarettes when they knew in the initial pilot that the cigarette tasted bad and no one would buy them.

Project Sizing

If a project has a good business case, then the next step is to figure out how big the project is.  This requires that you expand  the requirements sufficiently to estimate time and cost.

When it comes to software -- size matters.

Not only does size matter, you need to understand the level of uncertainty (requirements, technical, or skill) tied to your project to understand how much re-work or discovery will be necessary.

• Re-engineering a product with new technologies?  (high technical uncertainty)
• Building a new product with existing technologies? (high requirements uncertainty)
• Building software with a team unfamiliar with the technology? (have high skills uncertainty)

Skipping the time to get estimates is a major cause of project failure

There are still executives that give engineering a few hours or a few days to estimate major projects and wonder why the projects go late or get cancelled.

Estimate Uncertainty

Project methodology needs to take into account expected uncertainty.

Projects with low uncertainty can be handled with a traditional project methodology; these projects can have their work breakdown structures elaborated and dependencies can be worked out ahead of time.

Projects with moderate to high uncertainty will require rework and discovery.  This must be built into the project plan or you need to switch to an Agile methodology.  If you don't then there will be missing tasks on the project plan as well as task duration that is under-estimated.

Project tasks get stuck at 90% complete because the sources of uncertainty are not established a priori

Executing a project with personnel that are not competent with the technologies that you want to use is essentially professional malpractice.  If you must use existing personnel that don't know the technologies you want then you need to build in a large margin for training and rework over the project; these projects will take at least 2x as long as your worst estimate.

Focus on Early Defect Removal

Developer's say: "There is never enough time to write the code".  Statistics say, "There is never enough time to find all the defects" (see The Programmer Productivity Paradox).  To be successful you need to find defects as quickly as possible, especially since:

• If it costs X to find a defect pre-test 
• It costs 10X to find it in QA
• It costs 100X to find it once deployed in the field

You do the math, it costs less to find defects before they get to QA.  You are kidding yourself if you think that it is cost effective to have huge QA departments (see The Cost of Not Doing Quality).

Conclusion

Building software reliably and repeatably is not difficult.  What is difficult is getting organizations to realize that there is a minimum set of processes that must be followed to have a successful project.

Organizations regularly skip one or more steps and wonder projects fail to come together and succeed.