Waterfall is for Losers?

Perhaps that is a bit harsh, but in the question of whether Agile or Waterfall is best, the reality is that Agile ends up being multiple small waterfall cycles.  So you end up doing Waterfall in pieces.

All projects are composed of a set of tasks, typically executed by multiple people.  Within your set of tasks, some are dependent on others.  In particular we can never get away from the fact that to implement any single requirement, we need to analyse the requirement (what), design the solution (how), code the solution (execute), and test the result.

So any iterative and incremental project, i.e. Agile, Scrum, RAD, XP, etc.  Is really a series of small waterfalls.  The difference is that between the waterfalls we have sandwiched requirements, analysis, design, and coding which gives us the ability to change direction when we need to.

Note: If you never need to change direction (which is not often :-)) then Waterfall is a viable option.  Waterfall makes the most sense in projects where there is little to no requirements or technical uncertainty^*, e.g. reimplementing a system with experienced developers in a new technology where the use cases for the previous system are not changing.

The typical waterfall project assumes the above picture, that all requirements can be done then all design all coding and then all testing.  It is based on a 1960's factory mentality that code can be assembled like a car on an assembly line.

Waterfall only works if the amount of rework in any phase is immaterial and does not materially affect the length of the project.  Typically this is not true.  In the presence of requirements uncertainty, requirements need to be revisited many times.  Often, missing or inconsistent requirements will cause scope to change requiring change requests in the process.

All change requests will affect the project schedule.

In the presence of technical uncertainty, design needs to be revisited many times.  Often when using newer APIs the API is not documented or even worse, the code mechanism from the creator/vendor simply does not work and requires technical work arounds that materially affect the project schedule.

This will create a change request that will affect the project schedule.

An agile project builds requirements and design into every sprint.  This allows you to change directions when either requirements or technical decisions are uncertain.

This is the series of small waterfalls -- you are still doing Waterfall, but in an Agile way.

When you add up all the sprints of a project, you will realize that you have done the same project (i.e. the work is the work), but you have sliced it up differently.

So it is not a question of whether you will be doing Waterfall or not.  It is a matter of what is the sprint cycle that will support the requirements and technical uncertainty of your project.  If your project has little to no uncertainty, then your sprint cycle is the entire project.

If your project has requirements or technical uncertainty, then sprint cycles of 2-3 weeks are ideal.

^*Note: Similar to requirements and technical uncertainty are projects where there are significant dependencies on client interaction, so schedule uncertainty can also drive the need for agile project management.

---

Other articles in the "Loser" series

Want to see more sacred cows get tipped? Check out

• Comments are for Losers
• Defects are for Losers
• Debuggers are for Losers
• Efficiency is for Losers
• Testing departments are for Losers

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book once :-)

User stories or use cases?

Alistair Cockburn says "A user story is to a use case as a gazelle is to a gazebo".

If you are extremely familiar with both techniques then this makes sense.  If you are not familiar with both then suffice it to say that you can put a gazelle in a gazebo but not vice versa.

A user story is of the form As a <type of user>, I want <some goal> so that <some reason> (see here).  

A use case for withdraw money has many more components to it, and a skeleton of such a use case can be found here.  I develop this skeleton use case in a four part tutorial:

• Part 1: Use case as a dialog
• Part 2: What is an actor?
• Part 3: Adding interface details
• Part 4: Adding application context

The above user story is only a part of the entire use case.  For an ATM, a user story might be As a user, I want to withdraw money so that I can have physical cash.  This user story is only the summary of the withdraw money use case without the details.

User stories come from the Extreme Programming methodology. The assumption was that there will be a high degree of interaction between the developers, and the end customer and that QA will largely be done through test driven development.  It is important to realize that Extreme Programming does not scale.

Once your development team gets large, i.e. you have 3+ agile teams and your code base gets large enough to warrant a formal testing environment, then you will outgrow user stories as your only method of capturing requirements.

You can still use user stories for new modules developed with an end customer to take advantage of the light weight and rapid nature of user stories, but at some point those user stories should transition to use cases.

Unfortunately, there are quite a few ways to build use cases and not all of them are effective.  Alistair Cockburn's method (found here) is an excellent way to capture use cases for more sophisticated systems.  There is no doubt that use cases are heavier weight than user stories, but consider this:

• Use cases can more easily be turned into test cases by QA
• With use cases you more easily prove that you have all the requirements
• Use cases annotated with screens and reports can be used to collaborate with remote off site customers
• Well written use cases can easily be broken up into a sprint back log
• Use cases will work if you are not using Agile development methods
  

So don't forgo the speed and dynamic nature of user stories, just recognize that there are limits to user stories and that you will need to transition to use cases when your project team or application grows.

User Stories are Rarely Appropriate

All tools are useful when used appropriately, and User Stories are no different.

User stories are fantastic when used in small teams on small projects where the team is co-located and has easy access to customers.

User stories can quickly fall apart under any of the following situations:

• the team or project is not small
• the team is not in a single location
• customers are hard to access
• project end date must be relatively fixed

User stories were introduced as a core part of Extreme Programming (XP). Extreme Programming assumes you have small co-located teams; if you relax (or abandon) any of these constraints and you will probably end up with a process out of control.

XP, and hence user stories, works in high intensity environments where there are strong feedback loops inside the team and with customers:

• Individuals and Interactions over Processes and Tools
• Customer Collaboration over Contract Negotiation

User stories need intense intra-team / customer communication to succeed

User stories are a light-weight methodology that facilitates intense interactions between customers and developers and put the emphasis on the creation of code, not documentation. Their simplicity makes it easy for customers to help write them, but they must be complemented with timely interactions so that issues can be clarified.

Large teams dilute interactions between developers; infrequent communication leads to a lack of team synchronization. Most organizations break larger teams into smaller groups where communication is primarily via email or managers -- this kills communication and interaction.

Larger projects have non-trivial architectures. Building non-trivial architecture by only looking at the end user requirements is impossible. This is like having all the leaves of a tree and thinking you can quickly determine where the branches and the trunk should go.

User stories don't work with teams where intense interaction is not possible. Teams distributed over multiple locations or time zones do not allow intense interaction. You are delusional if you think regular conference calls constitute intense interaction; most stand-up calls done via conference degrade into design or defect sessions.

When emphasis is on the writing of code then it is critical that customers can be accessed in a timely fashion. If your customers are indirectly accessible through product managers or account representatives every few days then you will end up with tremendous latency.

Live weekly demos with customers are necessary to flush out misunderstandings quickly and keep you on the same page

User stories are virtually impossible to estimate. Often, we use user stories because there is a high degree of requirements uncertainty either because the requirements are unknown or it is difficult to get consistent requirements from customers.

Since user stories are difficult to estimate, especially since you don't know all the requirements, project end dates are impossible to predict with accuracy.

To summarize, intense interactions between customers and developers are critical for user stories to be effective because this does several things:

• it keeps all the customers and developers on the same page
• it flushes out misunderstandings as quickly as possible

All of the issues listed initially dilute the intensity of communication either between the team members or the developers and customers. Each issue that increases latency of communication will increase misunderstandings and increase the time it takes to find and remove defects.

So if you have any of the following:

• Large or distributed teams
• Project with non-trivial architecture
• Difficult access to customers, i.e. high latency
• High requirements uncertainty but you need a fixed project end-date

Then user stories are probably not your best choice of requirements methodology. At best you may be able to complement your user stories with storyboards, at worst you may need some light-weight form of use case.

---

Light-weight use case tutorial:

• (1 of 4) A use case is a dialog
• (2 of 4) Use case diagrams (UML)
• (3 of 4) Adding screens and reports
• (4 of 4) Adding minimal execution context

---

Other requirements articles:

• Shift Happens (long)
• Don't manage enhancements in the Bug Tracker
• When BA means B∪ll$#!t Artist

Agile tools do NOT make you Agile

Ask yourself the following questions:

• Do great golf clubs make you a great golfer?
• Does a formula one race make you an expert driver?
• Do great development tools make you an expert developer?

Unless you are delusional, you know that the answer is NO to all these questions.  An expert's performance can be dramatically improved with the right tools, but a beginner will not perform better with a great tool.

No matter how Agile-enabled tools like Version One and JIRA become, using either tool will not make you Agile unless you understand what processes make agile development work.

I recently finished a contract at a large retailer where they use JIRA for an enterprise integration project. It was interesting to hear them use terms like sprint and back log; interesting because there was no regular sprint cycle, and the back log was only the tickets in the JIRA system.  JIRA supports the notion of user stories (epics and stories) yet neither of these were being used correctly.

The problem isn't  JIRA; the problem was that they believed that their process was Agile because JIRA supports agile development.

Being agile is about following the Agile Manifesto.

Agile software development is not informality. Agile development has fewer formal practices than traditional waterfall development and those formal practices need to be adhered to (see Agility is not informality).  Agile development is light weight because it avoids activities that are unnecessary to the production of working code, not because it avoids rigour and formality.

There are implementations of Scrum and XP that WORK and are agile because they implement sound development processes.  However, there are also plenty of organizations that are implementing Scrum and XP incorrectly (see Does Agile Hide Development Sins?)

Agile does not dictate that you have a fixed development CYCLE, but you can definitely succeed with one.  What is required is that you break a project into multiple cycles where you iterate on development.
Agile does not dictate HOW you keep your requirements (back log or stories) but you must have an effective requirements process.  If your requirements process is broken then you will never succeed with agile.

The Agile Manifesto puts the priority on things critical to development:

• Individuals and interactions
• Working product
• Customer collaboration
• Responding to change

However, agile does not reject things in the right hand column, unless they get in the way of the above factors:

• Processes and tools
• Comprehensive documentation
• Contract negotiation
• Following a plan

In particular, don't get get caught up in Processes and tools, i.e. Version One and JIRA.  These tools can help you if you understand the core principles of agile development.  If you don't, then these tools will leave you worse off than before.  Unfortunately, the popularity of agile development has every tool maker scrambling to change their product to support agile.

Tools will never extract or synthesize quality requirements and build quality code.  The only way to get to proper understanding of a project is to put priority on individuals and interactions; people and communication are the only way to solve problems -- tools are a secondary concern.

Individuals and interactions OVER processes and tools

Agile development does not dictate a fixed development cycle, but it does require that any cycle must finish with a working product.  The emphasis always has to be production code and that is why periodic demonstrations of working code are essential.

To accomplish this you need to understand your requirements, whether you have a back log or user stories.  There are many times when properly written use cases can be used.  Either way the requirements need to be correct and consistent, they need to be what the customer need, i.e. avoid hiding behind excessive documentation.

Working Product OVER excessive documentation

There is no way to get the requirements correct unless you have a strong working relationship with the customer.  The customer often does not get the initial requirements correct and often development does not understand them.  Focus on customer collaboration is the key to mutual understanding.

This is where most projects fall apart because both parties have a tendency to focus on contract negotiation. Contracts are necessary, but focusing on limiting liability and protecting yourself will not yield a relationship that leads to working software.

 Customer Collaboration OVER contract negotiation

You have to understand that following a plan does not make sense when either requirements or technical uncertainty causes you to change development direction due to the need to respond to change.

It means making reliable estimates up front of all requirements, not just caving into pressure from upper management (see Why Senior Management Declared Deadlines lead to Disaster).  This often means adding time to the projected end of the project when new requirements are discovered or technical challenges force work arounds.  Unfortunately, many projects do not re-adjust the projected end date under these circumstances, which leads to a death march project (see Death March Calculus)

Responding to change OVER following a plan

Agile development is about producing quality software by understanding the principles of the Agile Manifesto.  Agile development is not about deluding yourself simply because you are using a tool that supports agile development.  You can implement a very solid light weight agile solution using only spreadsheets.

Focus on being Agile first, then go find a tool

Other things that are compatible with agile development:

• Comments are for losers
• Inspections are not optional
• Bug Tracker Hell and How to Get Out

Agility is not Informality

Agile Formality:A “  Mole ” of Software Engineering Practices

Agility is about following the Manifesto for Agile Software Development, not the sloppy and crazy implementations of Scrum, XP, or other home-grown schemes out there. Sometimes I feel like people are using Agility as an excuse to be informal. Two of the key principles of the Agile Manifesto are: Working software over comprehensive documentation Responding to change over following a plan

Somehow some developers have interpreted this as meaning that there are no formal processes to be followed. But just because you are putting the priority on working software does not mean that there can be no documentation; just because you are responsive to change doesn't mean that there is no plan.

Agile development when properly implemented can accomplish great things.  When it is just an excuse to be undisciplined and cover-up cowboy development then a so called 'Agile' process will work very poorly and the organization will just spin its wheels.

Agile development in general raises productivity by 18.2% and quality by 27.1%

Extreme programming (XP) specifically raises productivity by 12.8% and quality by 17.8%

Agile software development does have formality and it requires quite a bit of discipline to stick with it. A few formal practices include:

• Stand-up meetings
• Work burn down charts
• Sprint demos
• Simplicity
• Improvement through reflection

Stand-up Meetings
Stand-up meetings are an integral part of the Scrum process and is a key discipline of Scrum, when they are done properly. This meeting is a coordinating mechanism of a self-organizing team where each member states:

• What have they finished
• What are they working on
• What external issues do they need help on

The scrum daily meeting raises productivity by 16.4% and quality by 23.5%

Believe it or not, there are some Scrum teams that sit down for the stand-up meeting. What the heck is this? The purpose of standing up is to be slightly uncomfortable and keep in mind that the meeting is supposed to move quickly.

You state what features that you have finished so that team mates waiting on your deliverables know where you are at. Finished means that you have unit tested your code (i.e. TDD) and that black box testing can start if there is a tester in your Scrum team.

Automated unit testing raises productivity by 16.5% and quality by 23.8%

You state what you are working on so that other team mates know what you are doing and can organize their work accordingly.  Don't bother to state how you solved problems, that was an issue for planning with team mates before you started coding.

You state what you need help on that is external to the Scrum team so that the Scrum master can resolve the issue.  This is not a design meeting, if there are issues that you need to resolve with team mates then you can resolve this issue right after the meeting.

Burn Down Charts

Work burn down charts is a formal discipline and essential to success (see Who Needs Formal Measurement?).  The work burn down chart is the heart of accountability in Scrum.  For the chart hours to get to 0 on a regular basis, it relies on the principles of: 1) commitment, 2) execution, and 3) estimates.

The point of a self-organizing team is that it commits to work each sprint to get the work done.  If it turns out that they over committed to a sprint then they must do whatever it takes to get the job done.  That may mean extra hours or it may mean being extra creative, but once you commit to work then you have got to get it done.  That means execution, it means doing what you say that you will do.

You may have a sprint or two where the work burn down chart does not go to 0.  If this is happening on a regular basis then you have either an estimation problem or a team commitment problem.  If engineers are regularly underestimating their tasks then it means that you really don't know how long things take.  You should consider some of the techniques in the Personal Software Process (PSP) to improve your ability to estimate.

PSP development raises productivity by 21.3% and quality by 31.3%

Another cause of work burn down charts not going to 0 is that the team is not really self-organizing.  The team needs to commit to the work to be done, it will never work if the organization assigns the work to be done in the sprint.

If you lose discipline during the sprint and don't keep the work burn down charts up to date, then you are not tracking progress correctly.  Remember, you can't manage what you don't measure.

Inadequate progress tracking reduces productivity by 16.0% and quality by 22.5%

Sprint Demos

The sprint demo is not optional, it is a mandatory part of the process.  The sprint demo is to demonstrate near-production code and demonstrate a working system.  The team needs to put pressure on itself that the code must be complete by the end of the sprint; this forces a self-organizing team think through the amount of work that they can commit to.  No demo = no team commitment.

Remember the whole point of Agile development is to put the emphasis on working software.  If you are not producing working software every sprint, then what are you doing? Whatever you are doing it is not Agile development.

The Agile Manifesto puts emphasis on working software.  If your sprints are not producing working software then you are not doing agile development.

Simplicity

The best designs are the simplest designs because they are easier to understand and maintain.  In a previous article No Experience Required! I discuss how years of experience do not make for better programmers.  You might expect that the best developers have more years of experience, but 8 studies over 30 years show that this is not the case (you might even want to get rid of poor performers with many years of experience, but see the article :-) )

In fact two persistent facts about the differences between the best and worst developers is that:

• program execution speed about 10 to 1
• program size 5 to 1

Much of this results from simplicity. The best developers plan their code and write simple and straight forward routines. The are rewarded by having smaller programs that execute quickly. The worst developers do not plan their code and shoot from the hip. They write convoluted code that is much larger and has much worse execution time.

Agility is about simplicity. It is spending 80% of your time thinking and 20% of the time writing simple and elegant code. It is not about spending 100% of your time writing complex and convoluted code that even you can't understand.

Simplicity does not happen, it is planned; it takes tremendous effort to write simple code.  The best developers plan their code (see PSP above) and are continually refactoring their code to make designs simpler to understand and maintain.

Automated restructuring raises productivity by 8.0% and quality by 11.7%

Code refactoring raises productivity by 4.3% and quality by 6.7%

Improvement through Reflection

A key discipline in Agile development is to reflect on what you can do better as individuals and as a team. There is always room for improvement, there is often a better way to do things.  Professional developers are keen to keep learning and improving their craft, and the best way to do this is by reflecting on the sub-optimal choices that we have made in the past and trying to do better in the future.

Conclusion

Agile development is about following the Agile Manifesto, it is not about informality.  True agility is developed by having the discipline to be formal in many things:

• Team synchronization
• Learning to estimate
• Tracking progress through burn down charts
• Committing to create production code
• Planning simple code
• Improving by not making the same mistakes

Agility is about being disciplined and formal in processes that count

---

References

• Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.¹
• Vierie del Bianco and Stosic, Dragan.  Agile Formality: A "Mole" of Software Engineering Practices²

¹N.B. All productivity and quality percentages were derived over 15,000+ actual projects
²Reference courtesy of Michael Jessop

---

Articles in the "Loser" series

Moo?

Want to see sacred cows get tipped? Check out:

• Comments are for Losers
• Efficiency is for Losers
• Defects are for Losers
• Debuggers are for Losers
• Not Planning is for Losers
• Testing departments are for Losers

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)

Does Agile hide development sins?

There are really only two key principles in software development that every project tries to adhere to:

1. To be highly focused on building the right thing (effectiveness principle).
2. Deliver code with few defects by preventing them from getting into the code or removing them as quickly as possible.

To be succinct it means build to the correct requirements and minimize defects.



At project completion your code falls into one of the above four buckets.  The amount of code in the correct requirements section with no defects is what determines if the project is successful. If you adhere to principle 1 then there should be little or no code in the red section.  If you adhere to principle 2 then the yellow section will be minimized.  The application of these two principles is what leads to a successful project.

Software projects are still failing at an alarming rate (see Understanding your chances) because people are still not understanding these two principles.  Initially principle 1 is much more important than principle 2; after all, who cares if you build a defect free system if it doesn't do what is needed?

For better or worse, so called Agile development  has become synonymous with virtually any iterative and incremental software process and is generally assumed to be Scrum or XP by many.

The Good News

The Agile manifesto supports principle 1 with the following statements:

•  Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.
• Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.
• Working software is the primary measure of progress.

This leads us to focus on an incremental approach to building software where your development cycle is shorter from 2-6 weeks.  Each development cycle has a requirements phase where you can adjust your aim and make sure that you stay on track.

The Bad News

Shorter development cycles  where you adjust your aim is critical to building the right software system, especially if you are faced with technical or requirements uncertainty (see Uncertainty trumps Risk). The problem is that iterative and incremental development will do very little to prevent defects from getting into your code or to get them out quickly once they are created.

With organizations adopting the so called Agile methodologies (Scrum, XP, etc) they are building better software by getting a handle on principle 1, the problem is that they are doing very little to address principle 2, i.e. minimizing defects.

For example, in Scrum all the defects get put into the back log to be addressed at a future time.  How often do people consider the source of the defect or whether it could have been avoided in the first place?

Defects come from multiple sources and need to be addressed by different people. The table below is from Bug Tracker Hell and How to Get Out! where we detail the different kinds of defects, how often they occur, and who has to fix them.

Defect Role Category	Frequency	Role
Requirements defect	9.58%	BA/Product Management
Architecture or design defect	14.58%	Architect
Code defect	16.67%	Developer
Testing defect	15.42%	Quality Assurance
Documentation defect	6.25%	Technical Writer
Database defect	22.92%	DBA
Website defect	14.58%	Operations/Webmaster

Agile won't help you eliminate requirements defects!

With a incremental build cycle that focuses on requirements each cycle you are virtually guaranteed to make some kind of forward progress on your project. But, if you have a poor requirements process and/or poor business analysts (product managers) then you may take 2-3 cycles to solve a problem that could have been solved in one cycle.  The mechanics of a short development cycle will bury defects in your requirements.

Agile won't help you eliminate design defects!

If your development team has built a suboptimal architecture then defects in that architecture will be difficult to detect.  You know that you have an architectural defect when what seems to be a simple change has a huge time estimate on it, something that ends up touching many of the source files. Agile will cover up these problems as people incrementally build out work around solutions to fixing the architecture.  In reality, your architecture will get worse and worse over time and this will manifest in slower and slower development.

Agile won't help you eliminate testing defects!

Defects in test plans and test cases occur 15.4% of the time.  These defects manifest in your bug tracker as Functions as Designed defects.  Each of these defects masks the fact that QA either:

• did not get a requirement
• got a bad requirement for the defect
• did not understand the requirement correctly

Since QA is a continuous process with Agile, QA will fix the test case during the cycle in which they discover it.  This does not leave you with any information on how to prevent these kinds of defects from happening again. Just these 3 categories of defects account for 40% of defects in your development process, and the way that most organizations will implement "Agile" development will bury these problems.  It may surprise you to learn that...

True Agile development does not have these problems.

So just because you are doing Scrum, XP, Crystal, DSDM, or Lean and Kanban software development does not mean that you are Agile.

What it Means to Be Agile

To really be Agile you must support all the principles behind the Agile manifesto. The Agile manifesto supports principle 2 with the following statements:

• Simplicity--the art of maximizing the amount  of work not done--is essential.
• Continuous attention to technical excellence and good design enhances agility.
• At regular intervals, the team reflects on how  to become more effective, then tunes and adjusts its behavior accordingly.

One of the consequences of implementing the simplicity principle above is to prevent defects from getting into the system and causing work for someone else.  This means categorizing your defects and understanding when any of the extended team members (BAs, architects, development, and QA) is creating defects that sandbag your overall development. It means conducting inspections at the requirements, design, and code level to remove defects before they get into the system.  Organizations that do inspections at all these levels achieve an overall defect removal of 97%.  Organizations that do not do inspections will only be able to remove about 80-85% of their defects  (see Capers Jones and Olivier Bonsignour, "The Economics of Software Quality").

To really be Agile you must support all the principles behind the Agile manifesto.