Waterfall is for Losers?

Perhaps that is a bit harsh, but in the question of whether Agile or Waterfall is best, the reality is that Agile ends up being multiple small waterfall cycles.  So you end up doing Waterfall in pieces.

All projects are composed of a set of tasks, typically executed by multiple people.  Within your set of tasks, some are dependent on others.  In particular we can never get away from the fact that to implement any single requirement, we need to analyse the requirement (what), design the solution (how), code the solution (execute), and test the result.

So any iterative and incremental project, i.e. Agile, Scrum, RAD, XP, etc.  Is really a series of small waterfalls.  The difference is that between the waterfalls we have sandwiched requirements, analysis, design, and coding which gives us the ability to change direction when we need to.

Note: If you never need to change direction (which is not often :-)) then Waterfall is a viable option.  Waterfall makes the most sense in projects where there is little to no requirements or technical uncertainty^*, e.g. reimplementing a system with experienced developers in a new technology where the use cases for the previous system are not changing.

The typical waterfall project assumes the above picture, that all requirements can be done then all design all coding and then all testing.  It is based on a 1960's factory mentality that code can be assembled like a car on an assembly line.

Waterfall only works if the amount of rework in any phase is immaterial and does not materially affect the length of the project.  Typically this is not true.  In the presence of requirements uncertainty, requirements need to be revisited many times.  Often, missing or inconsistent requirements will cause scope to change requiring change requests in the process.

All change requests will affect the project schedule.

In the presence of technical uncertainty, design needs to be revisited many times.  Often when using newer APIs the API is not documented or even worse, the code mechanism from the creator/vendor simply does not work and requires technical work arounds that materially affect the project schedule.

This will create a change request that will affect the project schedule.

An agile project builds requirements and design into every sprint.  This allows you to change directions when either requirements or technical decisions are uncertain.

This is the series of small waterfalls -- you are still doing Waterfall, but in an Agile way.

When you add up all the sprints of a project, you will realize that you have done the same project (i.e. the work is the work), but you have sliced it up differently.

So it is not a question of whether you will be doing Waterfall or not.  It is a matter of what is the sprint cycle that will support the requirements and technical uncertainty of your project.  If your project has little to no uncertainty, then your sprint cycle is the entire project.

If your project has requirements or technical uncertainty, then sprint cycles of 2-3 weeks are ideal.

^*Note: Similar to requirements and technical uncertainty are projects where there are significant dependencies on client interaction, so schedule uncertainty can also drive the need for agile project management.

---

Other articles in the "Loser" series

Want to see more sacred cows get tipped? Check out

• Comments are for Losers
• Defects are for Losers
• Debuggers are for Losers
• Efficiency is for Losers
• Testing departments are for Losers

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book once :-)

Seriously. The Devil Made me do It!

Just as eternal as the cosmic struggle between good and evil is the challenge between our two natures. Religion aside, we have two natures, the part of us that:

• thinks things through; makes good or ethical decisions a.k.a. our angelic nature
• reacts immediately; makes quick but often wrong decisions a.k.a. our devil nature

Guess the powers that be left a bug in our brains so that it emphasizes fast decisions over good / ethical decisions.

Quite often we make sub-optimal or ethically ambiguous decisions under pressure

You decide...

---

Situation: Your manager comes to you and says that something urgent needs to be fixed right away. Turns out the steaming pile of @#$%$ that you inherited from Bob is malfunctioning again. 

Of course Bob created the mess and then conveniently left the company; in fact, the code is so bad that the work-arounds have work-arounds.

Bite the bullet, start re-factoring the program when things goes wrong.  It will take more time up front, but over time the program will become stable.  Start code inspections on the critical sections to proactively reduce defects (see Inspections are not Optional)

Find another fast workaround and defer the problem to the future.  Find a good reason why the junior member of the team should inherit this problem.

---

Situation: You've got a challenging section of code to write and not much time to write it.

Get away from the computer, think things through.  Get input from your peers, maybe they have seen this problem before. 

Then plan the pathways out and write the code once cleanly. Taking time to plan seems counter intuitive, but it will save time. (see Not Planning is for Losers)

Naw, just sit at the keyboard and bang it out already.  How difficult can it be?

---

Situation: The project is late and you know that your piece is behind schedule. However, you also know that several other pieces are late as well.

Admit that you are late and that the project can't finish by the deadline.  Give the project manager and senior managers a chance to make a course correction.

Say that you are on schedule but you are not sure that other people (be vague here) will have their pieces ready on time and it could cause you to become late.

This situation is also known as Schedule Chicken...

---

Situation: You have been asked to estimate how long a critical project will take. You are only been given a short time to come up with the estimate.

Tell the project manager that getting a proper estimate takes longer than a few hours. Without proper estimates the project is likely to be severely underestimated and this will come back to bite you and the project manager in the @$$.  (See Who needs Formal Measurement?)

Tell the project manager exactly the date that senior management wants the project to be finished by.  You know this is what they want to hear, why deal with the problem now? This will become the project manager's problem when the project is late.

---

The statistics show that we often don't listen to our better (angelic?) natures very often.

So when push comes to shove and you have to make a sub-optimal or less than ethical decision, just remember:

The devil made you do it!

Run into other common situations? email me

Are Debuggers Crutches?

Debuggers have become powerful tools, but like a drug have we become too dependent on them? Since poor developers spend 25 times more time in the debugger there is a likelihood that people are zoning out in the debugger instead of using other methods.

Defects are common, but they are not not necessary.  They find their way into code because:

• Code pathways are not planned
• Developers are inattentive when coding
• Developers do not understand the requirements

Defect correction is only possible if you understand the code pathways, debuggers are not the best way to do this

Debuggers are commonly used by developer's to understand a problem, but just because they are common does not make them the best way to find defects.  I'm not advocating a return to "the good old days" but there was a time when we did not have debuggers and we managed to debug programs.

Note: in embedded systems it is very hard to get feedback if you do not use a debugger (or hardware debugger).  This article is not addressed to embedded developers who rarely have an alternative to a debugger.

Avoid Defects

The absolute best way to remove defects is simply not to create them in the first place. You can be skeptical, but things like  the Personal Software Process (PSP) have been used practically to prevent 1 of every 2 defects from getting into your code.  Over thousands of projects:

The Personal Software Process increases productivity by 21% and increases code quality by 31%¹

A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. This huge waste could be cut in half if all developers focused on not creating defects in the first place.

Not only does the PSP focus on code planning, it also makes developers aware of how many defects they actually create.  Here are two graphs that show the same group of developers and their defect injection rates before and after PSP training.2

Before PSP training	After PSP training

Finding Defects

Using a debugger to understand the source of a defect is definitely one way.  But if it is the best way then why do poor developers spend 25 times more time in the debugger than a a good developer? (see No Experience Required!)

Poor developers spend a week in the debugger for every 2 hours that good developer does

No one is saying that debuggers do not have their uses.  However, a debugger is a tool and is only as good as the person using it.  Focus on tools obscures lack of skill (see Agile Tools do NOT make you Agile)

If you are only using a debugger to understand defects then you will be able to remove a maximum of about 85% of all defects, i.e. 1 in 7 defects will always be present in your code.

Would it surprise you to learn that their are organizations that achieve 97% defect removal³?  Software inspections take the approach of looking for all defects in code and getting rid of them. Learn more about software inspections and why they work here:

• Inspections are not Optional
• Software Professionals do Inspections

Software inspections increase productivity by 21% and increases code quality by 31% 

Even better, people trained in software inspections tend to inject fewer defects into code. When you become adept at parsing code for defects then you become much more aware of how defects get into code in the first place.

But interestingly enough, not only will developers inject fewer defects into code and achieve defect removal rates of up to 97%, in addition:

Every hour spent in code inspections reduces formal QA by 4 hours

Conclusion

As stated above, there are times where a skilled professional will use a debugger correctly.  However, if you are truly interested in being a software professional then:

• You will learn how to plan and think through code before using the keyboard
• You will learn and execute software inspections
• You will learn techniques like PSP which lead to you injecting fewer defects into the code

You are using a debugger as a crutch if it is your primary tool to reduce and remove defects

Related Articles

Want to see more sacred cows get tipped? Check out:

Comments are for Losers Efficiency is for Losers Debuggers are for Losers Testing departments are for Losers
	Moo?

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)

References

• Gilb, Tom and Graham, Dorothy. Software Inspections
• ¹Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008.
• ³Jones, Capers. The Economics of Software Quality. 2011
• Radice, Ronald A. High Quality
• ²Watts, Humphrey. Introduction to the Personal Software Process

Agile tools do NOT make you Agile

Ask yourself the following questions:

• Do great golf clubs make you a great golfer?
• Does a formula one race make you an expert driver?
• Do great development tools make you an expert developer?

Unless you are delusional, you know that the answer is NO to all these questions.  An expert's performance can be dramatically improved with the right tools, but a beginner will not perform better with a great tool.

No matter how Agile-enabled tools like Version One and JIRA become, using either tool will not make you Agile unless you understand what processes make agile development work.

I recently finished a contract at a large retailer where they use JIRA for an enterprise integration project. It was interesting to hear them use terms like sprint and back log; interesting because there was no regular sprint cycle, and the back log was only the tickets in the JIRA system.  JIRA supports the notion of user stories (epics and stories) yet neither of these were being used correctly.

The problem isn't  JIRA; the problem was that they believed that their process was Agile because JIRA supports agile development.

Being agile is about following the Agile Manifesto.

Agile software development is not informality. Agile development has fewer formal practices than traditional waterfall development and those formal practices need to be adhered to (see Agility is not informality).  Agile development is light weight because it avoids activities that are unnecessary to the production of working code, not because it avoids rigour and formality.

There are implementations of Scrum and XP that WORK and are agile because they implement sound development processes.  However, there are also plenty of organizations that are implementing Scrum and XP incorrectly (see Does Agile Hide Development Sins?)

Agile does not dictate that you have a fixed development CYCLE, but you can definitely succeed with one.  What is required is that you break a project into multiple cycles where you iterate on development.
Agile does not dictate HOW you keep your requirements (back log or stories) but you must have an effective requirements process.  If your requirements process is broken then you will never succeed with agile.

The Agile Manifesto puts the priority on things critical to development:

• Individuals and interactions
• Working product
• Customer collaboration
• Responding to change

However, agile does not reject things in the right hand column, unless they get in the way of the above factors:

• Processes and tools
• Comprehensive documentation
• Contract negotiation
• Following a plan

In particular, don't get get caught up in Processes and tools, i.e. Version One and JIRA.  These tools can help you if you understand the core principles of agile development.  If you don't, then these tools will leave you worse off than before.  Unfortunately, the popularity of agile development has every tool maker scrambling to change their product to support agile.

Tools will never extract or synthesize quality requirements and build quality code.  The only way to get to proper understanding of a project is to put priority on individuals and interactions; people and communication are the only way to solve problems -- tools are a secondary concern.

Individuals and interactions OVER processes and tools

Agile development does not dictate a fixed development cycle, but it does require that any cycle must finish with a working product.  The emphasis always has to be production code and that is why periodic demonstrations of working code are essential.

To accomplish this you need to understand your requirements, whether you have a back log or user stories.  There are many times when properly written use cases can be used.  Either way the requirements need to be correct and consistent, they need to be what the customer need, i.e. avoid hiding behind excessive documentation.

Working Product OVER excessive documentation

There is no way to get the requirements correct unless you have a strong working relationship with the customer.  The customer often does not get the initial requirements correct and often development does not understand them.  Focus on customer collaboration is the key to mutual understanding.

This is where most projects fall apart because both parties have a tendency to focus on contract negotiation. Contracts are necessary, but focusing on limiting liability and protecting yourself will not yield a relationship that leads to working software.

 Customer Collaboration OVER contract negotiation

You have to understand that following a plan does not make sense when either requirements or technical uncertainty causes you to change development direction due to the need to respond to change.

It means making reliable estimates up front of all requirements, not just caving into pressure from upper management (see Why Senior Management Declared Deadlines lead to Disaster).  This often means adding time to the projected end of the project when new requirements are discovered or technical challenges force work arounds.  Unfortunately, many projects do not re-adjust the projected end date under these circumstances, which leads to a death march project (see Death March Calculus)

Responding to change OVER following a plan

Agile development is about producing quality software by understanding the principles of the Agile Manifesto.  Agile development is not about deluding yourself simply because you are using a tool that supports agile development.  You can implement a very solid light weight agile solution using only spreadsheets.

Focus on being Agile first, then go find a tool

Other things that are compatible with agile development:

• Comments are for losers
• Inspections are not optional
• Bug Tracker Hell and How to Get Out

The Programmer Productivity Paradox

Programmers seem to be productive people.

You always see them typing at their desks; they chafe for meetings to finish so that they can go back to their desks and code. When asked, they will say that there is not enough time to produce the code, and the sooner they can start coding, the sooner they will be done.

So writing code must be the most important thing, correct?

If the average programmer writes about 50 lines of production code a day.  A 50,000 line program would take 1,000 man days to produce.  The same 50,000 line listing can be entered by a programmer at about 1,000 lines a day or about 50 man days.

So what the heck are the developers doing for the other 950 days?

Before addressing the apparent contradiction, lets make a simple observation. Many methodologies (RUP, XP, Agile, Waterfall, etc) and programming languages have been compared over thousands of projects and we have determined that programmers write between 325 and 750 lines of code (LOC) per month.

That means  the 1,000 LOC per month suggested above is optimistic¹.  Even if programmers do not average 50 lines of code per day, the following is clear².

• Methodology does not explain the apparent productivity gap
• No language accounts for the apparent productivity gap

The reality is that only a fraction of a developer's time is actually spent writing production code. What developers are really doing is trying different combinations of code until they finally find the combination of code that works.

Or more correctly, the combination that yields a solution that neither QA nor the business analyst complains about.

That is why developers that plan their code before coding tend to outperform other developers. Few developers plan their code, but surprisingly, years of experience does not teach developers to learn to plan.  Studies over 40 years show that developer productivity does not change with years of experience. (see No Experience Required!)

Years of experience do not lead to higher productivity

Interestingly enough, there are methodologies that have been around for a long time that emphasize planning code.  Watts Humphrey is the created of the Personal Software Process (PSP)³.  Using PSP has been measured to:

PSP can raise productivity by 21% and quality by 31%

If you are interested there are many other proven methods of raising code quality that are not commonly used (see Not Planning is for Losers).

If your developers at their keyboard and not planning at a white board then odds are that your productivity is not as high as it could be.

If developers are spending all their time trying different combinations of code until QA is satisfied, then it means that most of their time is really spent establishing correct pathways and removing defects.  If you think about it, not having the correct pathways is a defect, so:

The reality is that there is not enough time to find the defects in the quickly written code

Interested in reducing defects in a cost-effective way? See Defects are for Losers.

Bibliography

1	The The Mythical Man Month is even more pessimistic suggesting that programmers produce 10 production lines of code per day
2	Jones, Capers and Bonsignour, Olivier.  The Economics of Software Quality.  Addison Wesley.  2011
3	Watts, Humphrey.  Introduction to the Personal Software Process, Addison Wesley Longman. 1997

---

Articles in the "Loser" series

Want to see more sacred cows get tipped? Check out: Comments are for Losers Defects are for Losers Debuggers are for Losers Efficiency is for Losers
	Moo?

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)

No Business Case == Project Failure

A business case comes between a bright idea for a software project and the creation of the software project. 

• To - idea to have a project is born
• Tcheck - formal or informal business case
• Tstart - project is initiated
• Tend - project finishes successfully or is abandoned

Not all ideas for software projects make sense.  In the yellow zone above, between idea and project being initiated, some due diligence on the project idea should occur.  This is where you do the business case, even if only informally on the back of a napkin.

The business case is where you pause and and estimate  whether the project is worth it, i.e. will this project leave you better off than if you did not do it. For those who want precise definitions the project should be NPV +ve.  In layman's terms, the project should leave the organization better off on it's bottom line or at least improve skill levels so that other projects are better off.

Projects that do not improve skills or the bottom line are a failure.

Out of 10 software projects (see Understanding your chances):

• 3 are successful
• 4 are challenged, i.e. over cost, over budget, or deliver much less functionality
• 3 will fail, i.e. abandoned

This means that the base rate of success for any software project is only 3 out of 10. Yet executives routinely execute projects assuming that they can not fail, even though the project team may know that the project will be a failure from day 1.

Business cases give executives a chance to stop dubious projects before they start. (see Stupid is as Stupid Does) Understanding how formal the business case needs to be comes down to uncertainty. There are three key uncertainties with every project:

• Requirements uncertainty
• Technical uncertainty
• Skills uncertainty

When there is a moderate amount of uncertainty in any of these three areas then a formal business case with cash flows and risks needs to be prepared.

Requirements Uncertainty

Requirements uncertainty is what leads to scope shift (scope creep).

The probability of a project failing is proportional to the number of unknown requirements when the project starts (see Shift Happens).

Requirements uncertainty is only low for two particular projects:

1. re-engineering a project where the requirements do not change
2. the next minor version of a software project. 

For all other software projects the requirements uncertainty is moderate and a formal business case should be prepared.

Projects new to you have high requirements uncertainty.

Technical Uncertainty

Technical uncertainty exists when it is not clear that all requirements can be implemented using the selected technologies at the level of performance required for the project. Technical uncertainty is only low when you have a strong understanding of the requirements and the implementation technology.  Uncertainty and risk are two different animals (see  Uncertainty Trumps Risk in Software Development)

When there is only a moderate understanding of the requirements or the implementation technology then you will encounter the following problems:

• Requirements that get clarified late in the project that the implementation technology will not support
• Requirements that can not be implemented once you improve your understanding of the implementation technology

Therefore technical uncertainty is high when you are doing a project for the first time and requirement uncertainty is high.  Technical uncertainty is high when you are using new technologies, i.e. shifting from Java to .NET or changing GUI technology.

Projects with new technologies have moderate to high uncertainty.

Skills Uncertainty

Skills uncertainty comes from using resources that are unfamiliar with the requirements or the implementation technology.  Skills uncertainty is a knowledge problem.

Skills uncertainty is only low when the resources understand the current requirements and implementation technology.

Resources unfamiliar with the requirements will often implement requirements in a suboptimal way when requirements are not well written.  This will involve rework; the worse the requirements are understood the more rework will be necessary.

Resources unfamiliar with the implementation technology will make mistakes choosing implementation methods due to lack of familiarity with the philosophies of the implementation libraries.  Often after a project is complete, resources will understood that different implementation tactics should have been used.

Formal or Informal Business Cases?

An informal business case is possible only if the requirements, technical, and skills uncertainty is low.  This only happens in a few situations:

• Replacing a system where the requirements will be the same and the implementation technology is well understood by the team
• The next minor version of a software system

Every other project requires a formal business case that will quantify what kind of uncertainty and what degree of uncertainty exists in the project.  At a minimum project managers facing moderate to high uncertainty should be motivated to push for a business case (see Stupid is as Stupid Does).

Here is a list of projects that tend to be accepted without any kind of real business case that quantifies the uncertainties:

• Change of implementation technology
  • Moving to object-oriented technology if you don't use it
  • Moving from .NET to Java or vice versa
• Software projects by non-software companies
• Using generalists to implement technical solutions
• Replacing systems with resources unfamiliar with the requirements
  • Often happens with outsourcing

Projects with moderate to high risks and no business case are doomed to fail.

Related articles

• No Experience Necessary!
  • Surprisingly there is no correlation between years of experience and productivity.  This has been verified over 40 years.
• Stop it! No, Really Stop It!
  • 5 worst practices that software organizations commonly practice that they need to stop right away

What the Heck are Non-Functional Requirements?

If functional requirements are for the end-users (customers), then who are the non-functional requirements for?

Non-functional requirements are for those that install and maintain operational code, i.e. the help desk and operators

Every developer needs to be aware of what those non-functional requirements are and why operations personnel and help desk personnel are indirect  customers that are really  important.  There is no way for these guys to be unhappy and it not to back up into development as an urgent problem!

Functional Requirements

Functional requirements are baked into the code that developer's deliver (interpreted or compiled).   Events from input devices (network, keyboard, devices) trigger functions to convert input into output -- all functions have the form:

This is true whether you use an object-oriented language or not.

Non-functional requirements involve everything that surrounds a functional code unit.  Non-functional requirements concern things that involve time, memory, access, and location:

Performance	Server must be able to handle 100 transactions a second End user must have less than 1 second response
Availability	The service has to be available 99.99% of the time during the service hours
Capacity	During service hours the server must support 700 simultaneous users
Continuity	Service is resilient to disk, machine, and operational center failure
Security	Ensure that only people authorized to access the service can Ensure that data is never corrupted due to illegal activity or machine failure

I won't spend any time on performance because this is the non-functional requirement that everyone understands.

Non-functional requirements are slightly different between desktop applications and services; this article is focused on non-functional requirements for services.

If you have any knowledge of ITIL you will recognize that the highlighted items deal with the warranty of a service.  In fact, the functional requirements involve the utility of a service, the non-functional requirements involve the warranty of a service.

Availability

Availability is about making sure that a service is available when it is supposed to be available. Availability is about a Configuration Item (CI) in the environment of the operations center that specifies how the code is accessed.  Availability is decided independently of the code and is at best part of the Service Design Package (SDP) that is delivered to the operations department.

Developer's need to be aware of single-points of failure (i.e. services hard-coded to a specific IP) which causes fits in operations that are not running virtual machines (VM) that can have virtual IPs . The requirement to create code that is not reliant on static IPs or specific machines is a non-functional requirement.  Availability is simplified in operations if the code is resilient enough to allow itself to easily move (or be replicated) among servers.

Availability non-functional requirements include:

• Code to verify that customers are in their user windows
• Automatic installation of CI or mechanisms
• Ability to detect and prevent manual errors for a CI
• Ability to easily move code between servers

Capacity

Capacity is about delivering enough functionality when required.  If you ask a web service to supply 1,000 requests a second when that server is only capable of 100 requests a second then something will fail.

This may look like an availability issue, but it is caused because you can't handle the capacity requested.

Internet services can't provide enough capacity with a single machine and operations personnel need to be able to run multiple servers with the same software to meet capacity requirements.  The ability to run multiple servers without conflicts is a non-functional requirement. The ability to take a failing node and restart it on another machine or VM is a non-functional requirement.

Capacity non-functional requirements include:

• Ability to run multiple instances of code easily
• Ability to easily move a running code instance to another server

Continuity

Continuity involves being able to be robust against major interruptions to a service, these include power outages, floods or fires in an operational center, or any other disaster that can disrupt the network or physical machines.

Where availability and capacity often involve redundancy inside a single operation center, continuity involves geographic and network redundancy.  Continuity at best involves having multiple servers that can work in geographically distributed operation centers.  At worst, you need to be able to have a master-slave fail over model with the ability to journal transactions and eventually bring the master back up.

Continuity non-functional requirements include:

• Resilience of a code base to potential network outages, i.e. ability to retry transactions or find a new server
• Making sure that correct error messages are returned when physical failures are encountered, i.e. if the network is unavailable then don't give the end-user a message "Customer record has errors, please correct".
• Ability to recognize inconsistent data and not continue to corrupt data inside the database.

Security

Security non-functional requirements concern who has access to functions and preventing the integrity of data from being corrupted.

Where access is concerned, how difficult will it be for operations personnel or help desks to set up security for users?

Developer's build in different levels of access into their applications without considering how difficult it will be for a 3rd party (help desk or operations) to set-up end users.

Data integrity is another non-functional requirement.  Developer's need to consider how their applications will behave if the program encounters corrupted data due to machine or network failures.  This is not as important an issue in environments using RAID or redundant databases.

Security non-functional requirements include:

• Ease of a help desk to set up a new user on an application
• Ability to configure a user's rights to enable them only to access the functions that they have a right to
• Ensuring through data redundancy or consistency algorithms that data is not corrupted

When You Forget Non-Functional Requirements...

Commonly start-ups are so busy setting up their services that the put non-functional requirements on the back burner.  The problem is that there are non-functional requirements that need to be designed into the architecture when software is created.

For example, it is easy to be fooled into building software that is tied to a single machine, however, this will not scale in operations and cause problems later on.  One of the start-ups I was with built a server for processing credit/debit card transactions without considering non-functional requirements (capacity, continuity).

 It cost more to add the non-functional requirements than it cost to develop the software!

Every non-functional requirement that is not thought through at the inception of a project will often represent significant work to add later on.  Every such project is a 0 function point project that will require non zero cost!

Generally availability, capacity, and continuity is not a problem for services developed with cloud computing in mind.  However, there are thousands of legacy services that were developed before cloud computing was even possible.

If you are developing a new service then make sure it is cloud enabled!

Operations People are People Too

Make no mistake, operations and help desk personnel are fairly resourceful and have learned how to manage software where non-functional requirements are not addressed by the developers.

Hardware and OS solutions exist for making up for poorly written software that assumes single machines or does not take into account the environment that the code is running in, but that can come at a fairly steep cost in infrastructure.

The world has moved to services and it is no longer possible for developers to ignore the non-functional requirements involved with the code that they are developing.  Developer's that think through the non-functional requirements can reduce costs dramatically on the bottom line and quality of service being delivered.

The guys that run  operational centers and help desks are customers that are only slightly less important than the end-user.  Early consideration of the non-functional requirements makes their lives easier and makes it much easier to sell your software/services. It is no longer possible for competent developers to be unaware of non-functional requirements.

---

Other Articles

	No Experience Necessary	Counter-intuitive evidence why years of experience does not make developers more productive
	Shift Happens	Why scope shift on development projects is inevitable and why not capturing requirements at the start of a project can doom it to failure.
	Inspections are not Optional	Software inspections are intensive but evidence shows that for each hour of inspection you can reduce QA by 4 hours!

Size Matters

Some say that software development is challenging because of complexity. This might be true, but this definition does not help us find solutions to reduce complexity. We need a better way to explain complexity to non-technical people.

The reality is that when coding a project size matters.  Size is measured in the number of pathways through a code base, not by the number of lines of code. Size is proportional to the number of function points in a project.

There are many IT people that succeed with programs of a certain size and then fail miserably when taking on programs that are more sophisticated.  Complexity increases with size because the number of pathways increase exponentially in large programs.

Virtually anyone (even CEOs :-) ) can build a hello, world! application; an application that only has a single pathway through it and is as simple as you can get.  Some CEOs write the simple hello, world! program and incorrectly convince themselves that development is easy. Hello, world! only has a single pathway through it and virtually anyone can write it.

main() {
     printf( "hello, world" ); 
}

If you have an executive that can't even complete hello,world then you should take away his computer :-)

Complexity Defined

As programs get more sophisticated, the number of decisions that have to be made increase and the depth of the call tree increases.  Every non-trivial routine will have multiple pathways through it.

If your average call depth is 10 with an average of 4 pathways through each routine then this represents over 1 million pathways.  If the average call depth is 15 then it represents 107 million pathways. Increasing sophisticated programs have greater call depth than ever and distributed applications increase the call depth even because the call depth of a system is additive. This is what we mean by complexity; it is impossible for us to test all of the different pathways in a black box fashion.

Now in reality every combination of pathways is not possible, but you only have to leave holes in a few routines and you will have hundreds, if not thousands, of pathways where calculations and decisions can go wrong.

In addition, incorrect calculations or decisions higher up in the call tree can lead to difficult to find defects that may blow up much further away from the source of the problem.

What are Defects?

Software defects occur for very simple reasons, an incorrect calculation is performed that causes an output value to be incorrect.  Sometimes there is no calculation at all because input data is not validated to be consistent and that data is either stored incorrectly or goes on to cause incorrect calculations to be performed.

We only recognize that we have a defect when we see an output value and recognize that it is incorrect. More likely QA sees it and tells us that we are incorrect.

Basically we follow a pathway that is correct through nodes 1, 2, 3, 4, and 5.  At point 6 we make a miscalculation calculation, and then we have the incorrect values at points 7 and 8 and discover the problem at node 9. So once we have a miscalculation, we will either continue to make incorrect calculations or make incorrect decisions and go down the wrong pathways (where we will then make incorrect calculations).

Not all Defects are Equal

It is clear that the more distance there is between a miscalculation and its discover will make defects harder to detect.  The longer the call depth the greater the chance that there can be a large distance between the origin and detection, in other words:

Size Matters

Today we build sophisticated systems of many cooperating applications and the call depth is exponential with the size of the system.  This is what we mean by complexity in software.

Reducing Complexity

Complexity is reduced for every function where:

• You can identify when inconsistent parameters are passed to a function
• All calculations inside of a function are done correctly
• All decisions through the code are taken correctly

The best way to solve all 3 issues is through formal  planning and development.Two methodologies that focus directly on planning at the personal and team level are the Personal Software Process (PSP) and the Team Software Process (TSP) invented by Watts Humphrey.

Identifying inconsistent parameters is easiest when you use Design By Contract (DbC) , a technique that was pioneered by the Eiffel programming language. It is important to use DbC on all functions that are in the core pathways of an application.

Using Test Driven Development is a sure way to make sure that all calculations inside of a function are done correctly, but only if you write tests for every pathway through a function.

Making sure that all calculations are done correctly inside a function and that correct decisions are make through the code is best done through through code inspections (see Inspections are not Optional and Software Professionals do Inspections).

All techniques that can be used to reduce complexity and prove the correctness of your program are covered in Debuggers are for Losers.  N.B. Debuggers as the only formalism will only work well for systems with low call depth and low branching.

Conclusion

Therefore, complexity in software development is about making sure that all the code pathways are accounted for.  In increasingly sophisticated software systems the number of code pathways increases exponentially with the call depth. Using formal methods is the only way to account for all the pathways in a sophisticated program; otherwise the number of defects will multiply exponentially and cause your project to fail.

Only projects with low complexity (i.e. small call depth) can afford to be informal and only use debuggers to get control of the system pathways. As a system gets larger only the use of formal mechanisms can reduce complexity and develop sophisticated systems. Those formal mechanisms include:

• Personal Software Process and Team Software Process
• Design by Contract (via Aspect Oriented Programming)
• Test Driven Development
• Code and Design Inspections