The Art of War: How it Applies to Software

Sun Tzu wrote:

War is of vital importance to the state; hence it is a subject of inquiry which can on no account be neglected.

In our modern world, software is of vital importance to your organization.   If you can build software consistently and reliably you will gain a tremendous advantage over your competition.  

If developing software is like waging war then who is the enemy? 

The enemy is complexity.  Complexity comes from having to make dozens of decisions correctly for your software project to even have a chance of succeeding.  For every thing that you know at the outset of the project, there will be at least ten things that you don't know.   Complexity is compounded by making those decisions on a deadline, especially if that deadline is not well chosen.

So when you consider launching a software project you are looking at the tip of the iceberg.  Your ability to handle uncertainty will dictate how successful you will be.

Unlike a conventional war this enemy does not sleep, has no obvious weaknesses, and can not be deceived.  Once you engage the enemy your success will depend on correctly estimating the magnitude of complexity and making excellent decisions.

Sources of Uncertainty

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.

Attack by Stratagem, p. 18

Uncertainty comes from a few main sources:

• Team resources unfamiliar with the technologies to be used
• Requirements that are incomplete or inconsistent
• Technical requirements that turn out to be infeasible
• Inability to understand project dependencies
• Inability to formulate a correct and understandable plan

Unlike fighting a conventional enemy, complexity will find all your weaknesses.  The only way to defeat complexity is through understanding the requirements, having well trained teams, building a solid project plan, and executing well. 

Unfortunately the way most organizations develop software resembles the Charge of the Light Brigade (poem). 

In that battle about 400 men on horses attacked 20 battalions of infantry supported by 50 artillery pieces. 

Needless to say it was a slaughter.

The Art of War (online)

Several principles outlined by Sun Tzu apply to software development as well:

• Deception
• Leading to advantage
• Energy
• Using Spies
• Strengths and weaknesses
• Winning whole

Deception

Since the enemy is complexity we can not deceive it.  Rather the problem is that we allow ourselves to be deceived by complexity. 

How often do you see developer's say that they can code anything over a weekend over a case of Red Bull? 

People generally do not launch software projects because they want to fail; but when only 3 out of 10 projects succeed, it shows that people have been deceived about the complexity of building software.

This statistic has haunted us for 50 years.

Senior management should really pause and consider if all their ducks are lined up in a row before embarking on a software project.  Unfortunately senior management is still underestimating complexity and sending teams to face virtually impossible projects.

Leading to advantage

Thus it is that in war the victorious strategist seeks battle after the victory has been won, whereas he who is destined to defeat first fights and afterwards looks for victory in the midst of the fight.
                            Tactical Dispositions, p. 15

Complexity emerges from the sources of uncertainty mentioned above.  Successful organizations plan to remove known uncertainties and have a plan to handle the ones that will emerge.  Getting into a project before understanding the magnitude of the uncertainty is a recipe for failure.

Energy

That the impact of your army may be like a grindstone dashed against an egg—this is effected by the science of weak points and strong

Energy, p. 4

The team's energy needs to be directed against uncertainty with appropriate force at the appropriate time.  When this happens uncertainty will be minimized and the chances for success will increase.

It is imperative to create a table of all risks that might affect your software project.  If you aggressively minimize the probability of risks triggering you will reduce uncertainty and increase your chances of success.

Software projects succeed when there is a rhythm to the attack on complexity.  High intensity problem solving needs to be followed by lower intensity stability building.  The team must move at a sustainable pace or risk burning out.  Software teams do not succeed when they are working 10+ hours a day; they become like dull swords -- unable to do anything.

Using Spies

Foreknowledge cannot be elicited from ghosts and spirits; it cannot be inferred from comparison of previous events, or from the calculations of the heavens, but must be obtained from people who have knowledge of the enemy's situation.
                                                                                                    Using Spies, 5 —6

The enemy is complexity and is intangible, i.e. invisible, odorless, and untouchable. Your spies are your business analysts, architects, and project managers.

Your business analysts will work with the business to define the scope of the complexity.  Ask for anything you want, but commit to build all you ask!

Remember that all large complex systems that work are built from small simple systems that work, so aim to build the smallest usable product initially.  Asking for too much and providing insufficient resources and/or time will lead to a failed project.

The architects provide checks and balances on the business analysts to make sure that the project is feasible.

The architects will provide key dependency information to the project managers who make sure that a proper execution plan is created and followed.

Each of these spies sees a different aspect of complexity that is not visible to other people. Unless the reports of three types is combined effectively you risk not knowing the extent of the software that you are trying to build.  If you go into battle without proper intelligence you are back to the scenario of the Charge of the Light Brigade.

Strengths and weaknesses

Military tactics are like unto water; for water in its natural course runs away from high places and hastens downwards.  So in war, the way is to avoid what is strong and to strike at what is weak.
                                     Weak Points and Strong, p. 29—30

Water exhibits ordered flexibility.  It is ordered because it seeks to flow downhill; however, it is flexible and will go around rocks and other obstacles.  A software project needs to make continual progress without getting sandbagged with obstacles.  Methodologies like RUP or Agile software development can make sure that you exhibit ordered flexibility.

Winning whole

Winning whole in a software project means delivering the software on time and on budget without destroying the health and reputations of your team (including management).  Failed projects extend their effects to every member of the team and everyone's resume.

When you engage in actual fighting, if victory is long in coming, then men's weapons will grow dull and their ardor will be damped.
                                                               Waging War, p. 3

When organizations bite off more than they can chew they exert tremendous pressures on the team resources to work extended hours to make deadlines that are often unrealistic.  

In the pressure cooker you can expect key personnel to defect and put you into a worse position.  How many times have you found yourself on a Death March?

Conclusion

Both waging war and software development are serious topics that involve important struggles.  If software development is a war against ignorance, uncertainty, and complexity then many of the strategies and tactics outlined in The Art of War give us direction on how to execute a successful project. 

Uncertainty and Risk in Software Development (1 of 3)

To develop high quality software consistently and reliably is to learn how to master complexity.  You master complexity when you understand the different sources of uncertainty and the different risk characteristics of each uncertainty.  Uncertainties introduce delays in development as you attempt to resolve them.  Resolving uncertainties always involves alternative designs and generally affects your base architecture.  Poor architecture choices will increase code complexity and create uncertainty as future issues become harder to resolve in a consistent manner.

Confused?  Let's untangle this mess one issue at a time.

Uncertainty

The key principle here is that uncertainty will introduce delays in development.  Let's look at the average speed of development.  The Mythical Man Month conjectures that an average developer can produce 10 lines of production code per day regardless of programming language.  Let's assume for the sake of argument that todays developers can code 100 lines of code per day.  Development speed is limited because of meetings, changed and confused requirements, and bug fixing.  

Suppose we take a working 200,000 line program and print all the source code.  If we ask a programmer to type this code in again, they are likely to be typing at least 2,000 lines of code per day.  So to develop the program from scratch would have taken 2,000 man days, but to type it in again would only take 100 man days.

The time difference has to do with uncertainty.  The developer that develops the application from scratch faces uncertainty whereas the developer that types in the application faces no uncertainty. 

If you have ever done mazes you discover that to do the maze from the entry to the exit point involves making decisions, and this introduces delays while you are thinking.  However, try doing a maze from the exit back to the entry, you will find there are few decisions to make and it is much faster.  Fewer decisions from resolving uncertainty faster leads to fewer delays.

It is always faster to do something when you know the solution.

  

Sources of Uncertainty

The major sources of uncertainty are:

• Untrained developers
• Incomplete and inconsistent requirements
• Technical challenges

We use the term "learning curve" to indicate that we will be slower when working with new technologies.  The slope of the learning curve indicates how much time it will take to learn a new technology.  If you don't know the programming language, libraries/APIs, or IDE that you need to work with this will introduce uncertainty.  You will be constantly making syntax and semantic errors as you learn new languages, but this should pass rather quickly.  What will take longer is learning about the base functionality provided by the libraries/APIs. In particular, you will probably end up creating routines only to discover that they are already in the API.  Learning a new IDE can take a very long time and create serious frustration along the way!

Incomplete and inconsistent requirements are a big source of uncertainty.  Incomplete requirements occur when you discover new use cases as you create a system.  They also occur when the details required to code are unavailable, i.e. valid input fields, GUI design, report structure, etc.  In particular, you can end up iterating endlessly over GUI and report elements – things that should be resolved before development starts.  Inconsistent requirements occur because of multiple sources of requirements as well as poor team communication.

Technical challenges come in many forms and levels of difficulties.  A partial list of technical challenges includes:

• Vendor APIs that are poorly documented
• Vendor APIs that are buggy
• Interfacing incompatible technologies
• Insufficient architecture
• Performance problems 

In all cases technical challenge is resolved either by searching for a documented solution in publications, on the Internet, or by trial an error.  Trial and error can be done formally or informally but involves investigating multiple avenues of development, possibly building prototypes, and then choosing a solution.  While you are resolving a technical challenge your software project will not advance.

  

A common source of uncertainty is insufficient architecture.  Insufficient architecture occurs when the development team is not aware of the end requirements of the final software system.  This happens when only partial requirements are available and/or understood by the developers.  The development team lays down the initial architecture for the software based on their understanding of the requirements of the final software system.

Subsequently, clarified requirements or new requirements make developers realize that there was a better way to implement the architecture.  The developer and manager will have a conversation that is similar to:

Manager: We need to have feature X changed to allow behavior Y, how soon can we do this?

(pause from the developer)

Developer:  We had asked if feature X would ever need behavior Y and we were told that it would never happen.  We designed the architecture based on that.  If we have to have behavior Y it will take 4 months to fix the architecture and we would have to rewrite 10% of the application.

Manager:  That would take too long.  Look I don't want you to over engineer this, we need to get behavior Y without taking too much of a hit on the schedule.  What if we only need to have this for this screen?

(pause from the developer)

Developer:  If we ONLY had to do it for this one screen then we can code a work around that will only take 2 weeks.  But it would be 2 weeks for every screen where you need this.  It would be much simpler in the long run to fix the architecture.

Manager:  Let's just code the work around for this screen.  We don't have time to fix the architecture.

The net effect of insufficient requirements is that you end up with poor architecture.  Poor architecture will cause a technical challenge every time you need to implement a feature that the architecture won't support.  You will end up wasting time every time you need to work around your own architecture.

Management will not endorse the proper solution, i.e. fixing the architecture, because they have a very poor understanding that every work around that is made is pushing the project closer and closer to failure.  Eventually the software will have so many work arounds that development will slow to a crawl.

It is interesting that the project will probably fail, yet, soon enough the organization will attempt to build the same software using the same philosophy.

There is never enough time to get the project done properly, but there will always be enough time to do it again when the project fails.

http://www.geekherocomic.com/2009/06/03/clever-workaround/

Summary

• Uncertainty comes from several sources
• Untrained personnel (language, API, IDE)
• Inconsistent and incomplete requirements
• Technical challenges

Next part (2 of 3)

• Defining and understanding risk
• Matching uncertainties and risks

Understanding your chances of having a successful software project

We have been building software systems for over 50 years, and yet success rates remain extremely low, see Dan Galorath for some information.  Different reports put the success rates at different levels but successful projects are rarely higher than 30-40%.

Report	Year	Successful	Challenged	Failure
Standish Chaos Reports	2009	32%	44%	24%
Saur & Cuthbertson	2003	16%	74%	10%
Tata Consultancy	2007	38%	62%

It might seem strange, but we don't all have the same definition of success for software projects.  Success is when a project delivers the expected benefits within 10% of cost and schedule.  For me, a project is NOT successful when:

• It does not deliver what was promised
• It has cost or time over runs of over 10%
• It has its scope dramatically reduced so that victory can be claimed
• It does not have a positive net present value, i.e. it never breaks even

Under those conditions I'm guessing that there are even fewer successful projects out there.  Let's make software success extremely concrete.  Imagine that you are on a street corner watching people cross the street to the other street corner.  Imagine that out of every 10 people trying to cross the street only 3 people cross successfully, the other 7 get maimed or killed.

How interested would you be in crossing the street?

Sun Tzu wrote:

War is of vital importance to the state; hence it is a subject of inquiry which can on no account be neglected.

In our modern world, software is of vital importance to your organization.   If you can solve your business issues by building software consistently and reliably you will gain a tremendous advantage over your competition.

One misconception is that software projects fail because “I am surrounded by idiots!”   Just because we get frustrated at being unable to get software built does not make this statement true.  In fact, the exact opposite is true, the average IQ of Computer System Analysts is 111.3[1], and any IQ above 110 is considered to be Superior Intelligence[2]. 

Don't get me wrong; you might have a bunch of developers from the shallow end of the gene pool, but that does not explain how thousands of organizations fail to build quality software.  The point is that software does not fail because there are not enough smart people looking at the problem.

There are plenty of consultants, a.k.a. snake oil salesmen, that are willing to sell you “silver bullet” solutions that will solve your every problem.  Have you ever seen any of these really work?  Each of these solutions will generally solve one aspect of your problem and leave you with a larger one to fix later; they will also leave big holes in your budget.  Unfortunately, we often succumb to "silver bullet" solutions because they tell us what we want to hear.

To really fix your software development problems requires better understanding of basic principles; after all, there are software projects that succeed out there.  Not surprisingly, the companies that have figured out how to develop software consistently and reliably tend to have the fewest failures.  Learning how to develop software consistently and reliably requires that you learn how the following 5 elements intersect and affect each other:

1. Requirements
2. Project Management
3. Principles
4. Developers
5. Executives

Each of these elements will intersect with all of the others.  You will continually stumble through software development until you get the minimum level of execution and synchronicity between these 5 rings.

Organizations that do not understand these 5 rings will create organizational structures and processes that are doomed to fail.  Poor organizational structure and processes will create systemic problems that will lead to the following problems:

• Constant fire fighting (blog)
• Inflexible software (blog)
• Poor architecture (blog)

Problems from poor organizational structure and processes will lead to failed software projects because of the SYSTEM, not the PEOPLE.  However, people always assume that someone is to blame; they rarely look for problems inside the system.  This will lead to severe morale problems and the loss of competent personnel.

Fixing your software development is a matter of understanding the principles of good organizational and process design.  Once you understand how to balance the 5 elements you will begin to experience success in building software.

Cancellation vs Project Size

Function Points	Average Effort
10	1 man month
100	7 man month
1000	84 man month
10000	225 man year
100000	3197 man year

Appendix: Modern IQ Ranges for Various Occupations

According to modern IQ ranges, computer system analysts have one of the highest intelligence quotients of all professions.

---

[1] Average IQ by occupation (estimated from wordsum scores), January 22, 2011.  Available from http://anepigone.blogspot.ca/2011/01/average-iq-by-occupation.html

[2] What Different IQ Scores Mean, April 12, 2004.   Available from http://wilderdom.com/intelligence/IQWhatScoresMean.html

Basic UML – UML Diagrams – Extend use cases, Actor inheritance

In the last tutorial we introduced the idea of an included use case.  An included use case is a dialog sequence that occurs and will be invoked from one or more use cases.  An include use cases is always executed from the parent use case and represents mandatory behavior.

Extend Use Cases

The extend use case is also a dialog sequence; however, it is not invoked all the time.  The extend use case represents optional behavior, however, that is not a sufficient condition to create an extend use case.  In general, an extend use case must also have behavior that is not already handled by some other use case.  At this point our use case diagram looks as follows:

One common optional behavior for this use case would be Print out a gift receipt.  Not all purchases need to get a gift receipt and so this qualifies as optional behavior.   In addition, it is clear that a gift receipt will contain different information from the base receipt, i.e. no amounts will be shown, possibly a coupon will be printed.

We put the extend use case Print out a gift receipt and indicate that it is an extension to the Sell goods to customer use case.

Actor Inheritance

Let’s add a few more use cases for our Sales Representative:

• Return merchandise
• Find item in other stores

The updated diagram would be:

In addition, let's add the Manager actor to our system.  The manager in a retail system is able to do things like:

• Apply register discount
• Adjust inventory levels

The updated diagram would be:

The problem is that this diagram states that the Manager and the Sales Representative have mutually exclusive permissions to execute the use cases.  However this picture is incomplete, because in most retail stores, the Manager can also do everything that the Sales Representative can do.  A complete diagram would be as follows: 

This diagram is getting pretty crowded, and we have not even added all the retail use cases.  Fortunately, UML gives us a mechanism to easily indicate when one actor can do everything that another actor can do – Actor inheritance:

By designating the Manager as inheriting, or more precisely generalizing, the Sales Representative, we are declaring that the Manager can do everything that the Sales Representative can do.

This will go a long way towards cleaning up your use case diagrams.  Note, using actor inheritance does not express any new concepts on your diagram, it only simplifies the diagram.  To really simplify the diagram you would really create 2 diagrams in your UML tool:

Diagram 1:

Diagram 2:

Summary

• Use Include use cases to split out common use case dialog sections that repeat across use cases or are complicated enough to hit the purpose of your main use case
• Use Extend use cases to capture optional or exceptional behaviour to some use case
• Use actor Inheritance to simplify your diagrams by indicating when one actor can do everything that another actor can do.

Basic UML – UML Diagrams – Include use cases

The Universal Modeling Language (UML) is not the only way to produce diagrams for a software project, but it is the most accepted methodology.  The list of tools for UML is quite large and can be found here.  Please use the Internet to learn more about the history of UML and what it can do for you.

Recap of the Tutorials

Recently, we have focused on creating practical use cases.  Our tutorial on use cases showed that a use case is a dialog between an actor (role) and the system (tutorial 1 of 4).  We then explained what an actor was and introduced basic use case diagrams (tutorial 2 of 4).  The subsequent tutorials showed how to add screen information, data entry criteria, and reports to use cases to add detail (tutorial 3 of 4); and how to add context information to use cases so that it could be positioned within a project (tutorial 4 of 4).

The use case diagram shows a relationship between an actor and a use case.  A user is concrete but a role is abstract.  Just as any woman might have the role of a mother, daughter, girl friend, PTA member, etc in different contexts, a user may have one or more roles within a system.

Use cases are behavioral because they describe an interaction between a role and the system that will cause the system to change state.  So use case diagrams are functional. 

Use Case Diagram for a Retail Point of Sale System

Let's design software for a retail merchant.  The most basic thing that a merchant does is to sell goods to a customer.  The merchant sells those goods by using a sales representative and the Sell goods to customer use case might be as follows[1]:

Use Case Diagram

The simplest use case diagram is as follows:
 

This indicates that there is a role within the system called Sales Representative and that they have access to the Sell goods to customer use case.  The details of dialog between the sales representative and the system will be outlined in a text use case as shown above (MS Word).

Include Use Cases

An include use case includes a dialog sequence that is common to multiple use cases. An include use case can reduce the amount of writing in our use cases if two or more use cases must use the same dialog.  Ideally you write out both use cases completely and then make sure that a single section is identical. 

By combining common sequences into an included use case you not only reduce the amount of documentation but also reduce the chance of inconsistent requirements stemming from changes in one document that are not copied uniformly.

Include use cases can also be used if a sequence of dialog in a single use case requires so much documentation that it would obscure the essential purpose of the use case that it is part of.

To Include, or Not to Include

Creating an include use case when you really don't need it will increase your documentation and the chances of confusion during development.  Let's look at the sell goods to customer use case and see if there are any sequences that might be factored out into an include use case.

To sell goods to a customer the sales representative will enter one or more bar codes manually or via a scanner.  It seems possible that scanning a bar code might be a possible include use case.  If so it would change the diagram as follows:

  

We add a new use case to the diagram Scan bar code using scanner and designate Sell goods to customer as including it.  What this means is that the only way to invoke the dialog for Scan bar code using scanner is from the Sell goods to customer use case.  The use of the word include indicates that the functionality is necessary and will be invoked 100% of the time somewhere from the dialog for Sell goods to customer.

When we look closely at how bar code scanners work, they typically return the scanned bar code as a keyboard.  In other words, you can't really tell if the sales representative typed in the bar code or whether they scanned the bar code.  Since the behavior of the scanner does not create any alternate courses this would be an example of when NOT to use an include use case.

 

However, if you are developing the code to work with the scanner or the 3^rd party software returns multiple exceptions then each exception will probably trigger an alternate course.  The primary goal of Sell goods to customer might be obfuscated if the bulk of the use case turns out to be alternate courses dealing with the scanner.  In that case, it makes sense for Scan barcode using scanner to be a separate use case.

Only create separate use cases for inclusion under the following circumstances:

• The alternate courses created by one part of the dialog would obfuscate the primary goal of the use case
• Non-trivial development needs to be done on part of a dialog and you need to track that functionality separately in your bug tracking system

Print Receipt

Looking through the rest of the use case we see that the dialog required for step 6. System prints out a sales receipt could actually be quite complicated[2].

Clearly there will be variation of the receipt depending on whether the customer paid with cash or credit.  You will need to get the information about the sales representative and other details on the bill.

During development, there will probably be a developer assigned to handle all the coding for the receipt (and all the nit-picking changes that product management will make J).

The printing of the receipt is definitely going to involve more work than that involved in collecting the bar codes.  But before you create an include use case for Print out a sales receipt you should start by writing the dialog into the main use case Sell goods to customer.  If the use case turns out to be large and obscure the purpose of Sell goods to customer than it will deserve to have its own use case.

In general, most use cases that involve printing receipts and reports can be split out into separate use cases.

Summary

• Include use cases can be used to capture common dialogs that occur across multiple use cases
  
• Exceptionally use them when the alternate courses in a subset of a use case would cause the purpose of the entire use case to be obfuscated
• Try to create a single use case first, then split it up if it is too large
• Typically reports and receipts will get split out from the main use case

Next Steps

• In the next blog we consider extend use cases
  

---

[1] Note, we don’t show any alternate courses right now otherwise the use case would be quite long J

[2] Best Practices for Credit Card Transaction Receipts on http://bit.ly/9rWXul

Use Case - Alternative courses (2 of 2)

In our previous blog  we introduced the notion of alternative actions.  Alternative actions are deviations from standard behavior ranging from simple alternatives that do not disrupt the flow of the use case to full blown error exceptions.  How we document the alternative actions will depend on how that action affects the flow of the use case.

In the previous blog , we discussed alternative actions that could be handled inside the use case itself.  Now lets move on to alternative actions after state C.

State F

At this point the actor has successfully inserted his ATM card and it has been read.  We are now prompting the actor for his PIN. At this point we only have a few viable continuation paths:

1. The user enters a PIN
2. The user cancels the transaction
3. The ATM times out waiting for user input
   

The actor can enter his PIN in many different ways, but, between entering different digits and hitting the correction button most of these alternative actions do not change the flow of the use case.  We can handle this alternative by putting a table of valid inputs inside of the use case:

The user enters a PIN

Field	Type and Length	Comment
PIN	Numeric 4 digits	Digits to be entered with the keypad and will be drawn from left to right Digits will be masked with ‘*’ character
Correct	Button	Clears all entered key strokes
Ok	Button	Submits the form to the system

The user cancels the transaction (state G)

The actor canceling his transaction can certainly be handled inside of the use case by listing out all the alternative courses where the user can hit the cancel button.  However, the actor can hit the cancel in almost all use cases and to document all the alternative courses would be burdensome.

This kind of interruption in the flow of the use case is best documented on the use case diagram as a separate use case.  First we show the original use case diagram:

And we show virtually all of the use cases can be affected by the Cancel transaction functionality as follows:

An extended use case is functionality that extends an existing use case but is optional.

The Cancel Transaction use case would look as follows:

ATM times out (state Z)

An ATM time out can be handled the same way as the cancel transaction use case.  It should be in a separate use case that extends all use cases that have input, however, there are a few differences:

• This behavior is based on a system event, not a user event
• The user should be prompted on whether to continue or not, so the use case flow will depend on how the user responds to the prompt

 The ATM time out use case would look as follows:

Use cases

Use case	Actors	Type	Importance	Frequency
Authenticate user	Customer	Normal	Normal	1000 / day
Withdraw money	Customer	Normal	Very high	1000 / day
Cancel transaction	Customer	Extended	High	50 / day
Time out	Customer	Extended	High	100 / day

Summary

• Some alternative actions can be handled inside the use case, others are handled with separate use cases
• Optional use cases are shown by extensions on the use case diagram
• Alternatives during user input can be handled inside the use case
• When user inputs alter the flow of the use case or flow between the use cases it needs to be documented