Friday, 31 May 2013

Agility is not Informality

Agile Formality:A 
” of Software Engineering Practices
Agility is about following the Manifesto for Agile Software Development, not the sloppy and crazy implementations of Scrum, XP, or other home-grown schemes out there.

Sometimes I feel like people are using Agility as an excuse to be informal. Two of the key principles of the Agile Manifesto are:
  • Working software over comprehensive documentation
  • Responding to change over following a plan
Somehow some developers have interpreted this as meaning that there are no formal processes to be followed. But just because you are putting the priority on working software does not mean that there can be no documentation; just because you are responsive to change doesn't mean that there is no plan.

Agile development when properly implemented can accomplish great things.  When it is just an excuse to be undisciplined and cover-up cowboy development then a so called 'Agile' process will work very poorly and the organization will just spin its wheels.

Agile development in general raises productivity by 18.2% and quality by 27.1%

Extreme programming (XP) specifically raises productivity by 12.8% and quality by 17.8%

Agile software development does have formality and it requires quite a bit of discipline to stick with it. A few formal practices include:
  • Stand-up meetings
  • Work burn down charts
  • Sprint demos
  • Simplicity
  • Improvement through reflection
Stand-up Meetings
Stand-up meetings are an integral part of the Scrum process and is a key discipline of Scrum, when they are done properly. This meeting is a coordinating mechanism of a self-organizing team where each member states:
  • What have they finished
  • What are they working on
  • What external issues do they need help on
The scrum daily meeting raises productivity by 16.4% and quality by 23.5%

Believe it or not, there are some Scrum teams that sit down for the stand-up meeting. What the heck is this? The purpose of standing up is to be slightly uncomfortable and keep in mind that the meeting is supposed to move quickly.

You state what features that you have finished so that team mates waiting on your deliverables know where you are at. Finished means that you have unit tested your code (i.e. TDD) and that black box testing can start if there is a tester in your Scrum team.

Automated unit testing raises productivity by 16.5% and quality by 23.8%

You state what you are working on so that other team mates know what you are doing and can organize their work accordingly.  Don't bother to state how you solved problems, that was an issue for planning with team mates before you started coding.

You state what you need help on that is external to the Scrum team so that the Scrum master can resolve the issue.  This is not a design meeting, if there are issues that you need to resolve with team mates then you can resolve this issue right after the meeting.

Burn Down Charts
Work burn down charts is a formal discipline and essential to success (see Who Needs Formal Measurement?).  The work burn down chart is the heart of accountability in Scrum.  For the chart hours to get to 0 on a regular basis, it relies on the principles of: 1) commitment, 2) execution, and 3) estimates.

The point of a self-organizing team is that it commits to work each sprint to get the work done.  If it turns out that they over committed to a sprint then they must do whatever it takes to get the job done.  That may mean extra hours or it may mean being extra creative, but once you commit to work then you have got to get it done.  That means execution, it means doing what you say that you will do.

You may have a sprint or two where the work burn down chart does not go to 0.  If this is happening on a regular basis then you have either an estimation problem or a team commitment problem.  If engineers are regularly underestimating their tasks then it means that you really don't know how long things take.  You should consider some of the techniques in the Personal Software Process (PSP) to improve your ability to estimate.

PSP development raises productivity by 21.3% and quality by 31.3%

Another cause of work burn down charts not going to 0 is that the team is not really self-organizing.  The team needs to commit to the work to be done, it will never work if the organization assigns the work to be done in the sprint.

If you lose discipline during the sprint and don't keep the work burn down charts up to date, then you are not tracking progress correctly.  Remember, you can't manage what you don't measure.

Inadequate progress tracking reduces productivity by 16.0% and quality by 22.5%

Sprint Demos
The sprint demo is not optional, it is a mandatory part of the process.  The sprint demo is to demonstrate near-production code and demonstrate a working system.  The team needs to put pressure on itself that the code must be complete by the end of the sprint; this forces a self-organizing team think through the amount of work that they can commit to.  No demo = no team commitment.

Remember the whole point of Agile development is to put the emphasis on working software.  If you are not producing working software every sprint, then what are you doing? Whatever you are doing it is not Agile development.

The Agile Manifesto puts emphasis on working software.  If your sprints are not producing working software then you are not doing agile development.

The best designs are the simplest designs because they are easier to understand and maintain.  In a previous article No Experience Required! I discuss how years of experience do not make for better programmers.  You might expect that the best developers have more years of experience, but 8 studies over 30 years show that this is not the case (you might even want to get rid of poor performers with many years of experience, but see the article :-) )

In fact two persistent facts about the differences between the best and worst developers is that:
  • program execution speed about 10 to 1
  • program size 5 to 1
Much of this results from simplicity. The best developers plan their code and write simple and straight forward routines. The are rewarded by having smaller programs that execute quickly. The worst developers do not plan their code and shoot from the hip. They write convoluted code that is much larger and has much worse execution time.

Agility is about simplicity. It is spending 80% of your time thinking and 20% of the time writing simple and elegant code. It is not about spending 100% of your time writing complex and convoluted code that even you can't understand.

Simplicity does not happen, it is planned; it takes tremendous effort to write simple code.  The best developers plan their code (see PSP above) and are continually refactoring their code to make designs simpler to understand and maintain.

Automated restructuring raises productivity by 8.0% and quality by 11.7%

Code refactoring raises productivity by 4.3% and quality by 6.7%

Improvement through Reflection
A key discipline in Agile development is to reflect on what you can do better as individuals and as a team. There is always room for improvement, there is often a better way to do things.  Professional developers are keen to keep learning and improving their craft, and the best way to do this is by reflecting on the sub-optimal choices that we have made in the past and trying to do better in the future.

Agile development is about following the Agile Manifesto, it is not about informality.  True agility is developed by having the discipline to be formal in many things:
  • Team synchronization
  • Learning to estimate
  • Tracking progress through burn down charts
  • Committing to create production code
  • Planning simple code
  • Improving by not making the same mistakes
Agility is about being disciplined and formal in processes that count

1N.B. All productivity and quality percentages were derived over 15,000+ actual projects
2Reference courtesy of Michael Jessop

Articles in the "Loser" series

Want to see sacred cows get tipped? Check out:
Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)

Thursday, 30 May 2013

Who needs Formal Measurement?

We all know the expression "You can't manage what you can't measure", but do we really understand it?

After execution, feedback is an essential part of all processes.  Just think about how difficult it would be to drive from home to work wearing a blindfold.  Without your sense of sight to give you feedback on the traffic signals and the locations of other cars you would crash your car.  Yet we develop software systems without instituting formal measurement programs all the time and wonder why we succeed so rarely?  (for success rates see Understanding your chances of having a successful software project)

You can't manage what you can't measure

No measurement means no feedback, which means your chances of success are minimized. Success is possible without formal measurement but it is much easier with formal measurement.

Formal measurement raises productivity by 20.0% and quality by 30.0%

A best practice is one that increases your chance of succeeding, it does not guarantee it. It has been established that formal measurement is a best practice, so why do so few people do it?

Measurement has a cost and organizations are petrified of incurring costs without incurring benefits. After all what if you institute a measurement program and things don't improve?  In some sense managers are correct that measurement programs cost money to develop and unless measurement is executed correctly it will not yield any results.

But is there a downside to avoiding measurement?

Inadequate progress tracking reduces productivity by 16.0% and quality by 22.5%

Failure to estimate requirements changes reduces productivity by 14.6% and quality by 19.6%

Inadequate measurement of quality reduces productivity by 13.5% and quality by 18.5%

So there are costs to not having measurement.  Measurement is not optional, measurement is a hygiene process, that is, essential to any process but especially to software development where the main product is intangible.

A hygiene process is one which can prevent very bad things from happening. Hygiene processes are rarely fun and take time, i.e. taking a shower, brushing your teeth, etc.

History has show that it is much more cost effective to execute a hygiene process than take a chance of something very bad from happening, i.e. disease or your teeth falling out.

There are hygiene practices that we use every day in software development without even thinking about it:
  • Version control
  • Defect tracking
Version control is not fun, tracking defects is not fun, but the alternative is terrible.  Only the most broken organizations think that they can develop software systems without these tools.  These tools are not fun to use and virtually everyone complains about them, but the alternative is complete chaos.

Formal measurement is a best practice and a hygiene practice

The same way that developers understand that version control and defect tracking is necessary, an organization needs to learn that  measurement is necessary.

Is Formality Necessary?
The reality is that informal measurement is not comprehensive enough to give consistent results. If measurement is informal then when crunch time comes then people will stop measuring things when you need the data the most.

When you don't have enough formality then processes take longer and by extension cost more.  When you have too much formality then you have process for processes sake and things will also take a long time.  Any organization that implements too much formality is wasting their time, but so is any organization that does not implement enough.

When you suggest any formal process people immediately imagine the most extreme form of that process; which would be ridiculous if it is implemented that way. We have all been in organizations that implement processes that make no sense, but without measurement how do you get rid of these processes that make no sense?

For every formal process that makes sense, there is a spectrum of implementations.  The goal is to find the minimum formality that reduces time and costs.  When you find the minimum amount of formal measurement you will accelerate your development by giving yourself the feedback that you need to drive your development.

What to Measure
It seems obvious, but incorrect measurement and/or poor execution leads to useless results.  For example, trying to measure productivity by measuring the hours that the developers sit at their machines is as useful as measuring productivity by the number of cups of coffee that the developers drink.

Another useless measure is lines of code (LOC), in fact, Capers Jones believes that anyone using LOC as a measurement should be tried for professional malpractice!

Measuring the the three things mentioned above will improve productivity and quality because there will not be a negative effect on your organization:
  • measuring progress tracking (productivity +16.0%, quality +22.5%)
  • estimating requirements changes  (productivity +14.6%, quality +19.6%)
  • measurement of quality (productivity +13.5%, quality +18.5%)
Other things to measure are:
  • Activity based productivity measures (productivity + 18.0% , quality by 6.7%
  • Automated sizing tools (function points)  (productivity + 16.5% , quality by 23.7%
  • Measuring requirement changes  (productivity + 15.7% , quality by 21.9%

So to answer the question: who needs formal measurement?
We all need enough formal measurement to get the job done

N.B. All productivity and quality percentages were derived over 15,000+ actual projects

Articles in the "Loser" series


Want to see sacred cows get tipped? Check out:
Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)

Wednesday, 29 May 2013

Not Planning is for Losers

Only the ignorant don't plan their code pathways before they write them; perhaps only classes with getter and setter routines don't need to be planned.

The total number of pathways through a software system grow so quickly that it is very hard to understand their total number.

If a function X() with 9 pathways calls function Y() which has 11 pathways then the composition function XY() will have up to 9 x 11 = 99 possible pathways.

If function Y() calls function Z() with 7 pathways, then XYZ()  will have up to 693 = 9 x 11 x 7 pathways. The numbers add up quickly, for example a call depth of 8 functions each with 8 pathways means 10.7 million different paths; the number of possible pathways in a system is exponential with the total depth of the call tree.

Programs, even simple ones, have hundreds if not thousands (or millions) of pathways through them.

Negative vs Positive Assurance

Quality assurance can only come from the developers, not the testing department.

Testing is about negative assurance, which states that "I don't see anything wrong"; it doesn't mean that everything is correct, just that they can't find a problem.

Positive assurance which is guaranteeing that the code will execute down the correct pathways and only the developer can do that.

Quality assurance comes from adopting solid practices to ensure that code pathways are layed down correctly the first time

Any Line of Code can be Defective

If there are 10 pathways through a function then there there must be branching statements  to be able to direct program flow down each of the pathways. Each pathway may compute variables that may be used in calculations or decisions downstream. Clearly, each downstream function can potentially have its behavior modified by any upstream calculation.

When code is not planned then errors may cause execution to compute a wrong value.  If you are unlucky that wrong value is used to make a decision which may send the program down the wrong pathway.

If you are really unlucky you can go very far down the wrong pathways before you even identify the problem.

If you are really, really, really unlucky not only do you go down the wrong pathway but the data gets corrupted and it takes you a long time to recognize the problem in the data.

It takes less time to plan code and write it correctly than it takes to debug complex pathways.

Common Code Mistakes

Defects are generally caused either because of one of the following conditions:
  1. incorrect implementation of an algorithm
  2. missing pathways
  3. choosing the wrong pathway based on the variables
1) Incorrect algorithms will compute a wrong value based on the inputs.  The damage is local if variables are not involved, however, if the value is computed in a variable X then damage can happen throughout the scope that X is available. Example, bad decision at node 1 causes execution to flow down path 3 instead of 2.

2) Missing pathways have to deal with conditions.  If you have 5 business conditions and only 4 pathways then one of your business conditions will go down the wrong pathway and cause problems until you detect the problem.  Example, there was really 5 pathways at node 1, however, you only coded 4.

3a) The last problem is that the base values are correct but you select the wrong pathway.  This can lead to future values being computed incorrectly.  Example: at node 10 you correctly calculate that you should take pathway 11 but end up going down 12 instead.

3b) You might also select the wrong pathway because insufficient information existed at the time that you needed to make a decision.  Example:  insufficient information at node 1 causes execution to flow down path 3 instead of 4.

The last two issuses (2 or 3) can either be a failure of development or of requirements.  In both cases somebody failed to plan...

When Incorrect Code Gets to QA

When code with poor pathways gets to QA you have the following situation.  The program cruises down to point 5 where a calculation is wrong or an incorrect decision is taking which then leads to bad calculations. 

However, you are only going to notice the problem when:
  • Some tangible element is changed (i.e. screen, database, or report)
  • Either QA or a customer notices the problem which actually occurs at point 9
The longer it takes for someone to notice the defect or the further that point 5 is from point 9 in the code -- the longer and harder it will be to locate and fix the defect.

Defects that persist for a long time end up generating work-arounds in the code that can break when the original defect is repaired.  

In fact, 1 out of 7 defects being fixed will create a new one!

This is why planning is critical, if it costs X effort to find the defect in development, it will cost 10 X effort to find the defect in QA, and 100 X to find the defect if your customer notices it.

What if it is too late to Plan

Whenever you are writing a new section of code you should plan the code before you write it.  Existing code can take advantage of inspections to locate and remove defects.  Don't wait for defects to develop, proactively inspect all code, especially in buggy modules and fix all of the code pathways.

Code inspections can raise productivity by 20.8% and quality by 30.8%

Code Solutions

The Personal Software Process (PSP) proposes that every code section should be planned before it is implemented.  That means that you plan your code pathways using paper or a white board before using the keyboard.  Ideally, you would spendpart of your day planning with your colleagues on how best to write your code pathways.

PSP can raise productivity by 21.2% and quality by 31.2%

If you insist on writing code at the keyboard then you can use pair programming to reduce errors.  By having a second pair of eyes looking at your code algorithmic mistakes are less likely and incorrect decisions for conditions are looked at by two people; unfortunately, pair programming is not cost effective.

Pair Programming can raise productivity by 2.7% and quality by 4.5%

Studies confirm that code sections of high cyclomatic complexity have more defects than other code sections.   At a minimum, any code section with  high cyclomatic complexity should be planned by two or more people.  If this is not possible, then reviewing sections of high cyclomatic complexity can reduce downstream defects.

Automated cyclomatic complexity analysis can raise productivity by 14.5% and quality by 19.5%

Design Solutions

All large software projects benefit from planning pathways at the macroscopic level.  The design or architectural planning is essential to making sure that the lower level code pathways will work well.

Formal architecture for large applications can raise productivity by 15.7% and quality by 21.8%

Requirements Solutions

Many pathways are not created in development but are an artifact of the requirements.  If there is insufficient information to choose a proper pathway or there are insufficient pathways indicated then this is a failure of requirements.  Here are several techniques to make sure that the requirements are not the problem.

Joint application design (JAD) brings the end-users of the system together with the system architects to build the requirements.  By having end-users present you are unlikely to forget a pathway and by having the architects present you can put technical constraints on the end-users wish list for things that can't be built. The resulting requirements should have all pathways properly identified along with their conditions.

Joint application design can raise productivity by 15.5% and quality by 21.4%

Requirements inspections are the best way to make sure that all necessary conditions are covered and that all decisions that the code will need to make are identified before development.  Not inspecting requirements is the surest way to discovering that there is a missing pathway or calculation after testing.

Requirement inspections can raise productivity by 18.2% and quality by 27.0%

Making sure that all pathways have been identified by requirements planning is something that all organizations should do.  Formal requirements planning will help to identify all the code pathways and necessary conditions, however, formal requirements planning only works when the business analysts/product managers are skilled (which is rare :-( ).

Formal requirements analysis can raise productivity by 16.3% and quality by 23.2%

Other articles in the "Loser" series

Want to see more sacred cows get tipped? Check out
Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)


Wednesday, 8 May 2013

Why Management Declared Deadlines Lead to Disaster

Senior management, in the face of market pressure, will sometimes initiate a project to address a business need that has a management declared deadline.

Management will do this when they feel market pressures to create a new software product or version because of some impending business event in the market.

These market driven deadlines occur for a variety of reasons:
  • Competitor is releasing a new product/version that will eat your market share
  • Laws have changed that require compliance by your software system
  • Merger with another organization requires software systems to be integrated
  • You are a start-up and need to build version 1.0 to obtain financing
  • Sales has sold vapor-ware and engineering now needs to produce it
Some organizations (not many) are enlightened enough that they determine project size in function points and use automated sizing tools to estimate time and cost.  Large organizations have measured automated tools against actual results and determined that automated sizing tools estimate projects to within 5% of time and cost.

Counting function points will increase productivity by 17.5% and code quality by 25.8%.

Automated sizing tools will increase productivity by 16.5% and code quality by 23.7%.

However, on a regular basis automated sizing tools do not produce estimates that management likes.  So management will be adamant that the deadline is fixed and must be met at all costs. Some management groups are more dysfunctional than others about insisting that deadlines because of business reasons and not by estimates.

Rejection of estimates for business reasons will decrease productivity by 15.7% and code quality by 21.7%.

The most dysfunctional management will expect all existing project not to be impacted by the addition of a high priority project. They will say silly things like 'You will just have to multi-task' or something equally ridiculous.

Anyone who has worked in a multitasking mode is familiar with  Little’s law or that productivity drop-off that comes from trying to do multiple things at the same time.

Less dysfunctional management will realize that the high priority project will cause other projects to be late, but they will continue to expect that everything progresses even though resources will be committed to multiple projects. Project deadlines typically get set in the short term, which means that the resource for the project will be fixed.

Even if the funding is available to hire resources, the new resources will not be familiar enough with the corporate context to be of much use. The scope and quality of the project will be dictated by the market pressure that generated the project.

This project will have fixed scope, quality, time, and resources which will lead to a feasible project only once in a million projects.

Those familiar with project management know that you can only set 2 parameters of time, resources, and scope and then the 3rd parameter must be allowed to vary, if you want a feasible solution.

If you specify the resources and scope then the time must be allowed to vary; if you specify resources and time, then the scope must be allowed to vary. When management sets up a project like this the only practical variable that can stretch is time through resources working overtime.  Overtime = excessive schedule pressure.
Excessive schedule pressure will decrease productivity by 16.0% and code quality by 22.5%.

If an organization is well run and a management declared project occurs rarely then the extra stress due toregular basis. Such senior management teams are unaware that they are increasing risk by an order of magnitude and are always mystified when they are unable to achieve the results that they want.
working long hours can be handled.

Unless the senior IT manager has the courage to point out the problems that such a project will cause, the IT department is doomed to become a pressure cooker. Even if the IT manager has the courage he needs to have the disciple to refuse the pressure from the other senior managers until a feasible project is defined.

Unfortunately, most senior IT managers have neither the courage of their convictions nor the discipline to weather the storm.

The end result will lead to a pressure cooker in engineering, where the engineers are being asked to perform super human efforts for which they will get very little in return.

This kind of environment if prolonged will lead to your better engineers quitting which will cause all the rest of your projects to get even later. With the loss of productivity that comes from trying to multi-task, the code that is produced is guaranteed to have more bugs than normal, which will then cause the problem to flow into QA.  Even more insanity will follow when QA time is reduced or eliminated, after all, who needs QA?

Loss of key personnel will decrease productivity by 15.7% and code quality by 21.7%.

Needless to say, when faced with such time sensitive projects the work done on the requirements is minimal and so the end product (even if delivered) will generally not satisfy the market pressure the project was designed to alleviate. In short, when senior management responds to market pressures by declaring a project and its deadline it is virtually guaranteed to fail.

How should senior managers deal with market pressures?

When faced with market pressure senior management must be absolutely certain that the pressure is real. There are many situations where managers have declared that a project needs to be finished by a given time, puts too much pressure on the project, and then the project is late or canceled.

Some managers make everything an emergency in order to get their way, they often say 'We have no choice'.

In these situations the manager will push for a ridiculous date that will be missed.  They will then turn around and accept a much later date for the the project; if this is the case then the project was not real and management was not responding to a real pressure. This happens regularly in environments where politics is more important than getting things done.

If the project must be done (i.e. compliance project) then it will be critical to assess complete and consistent requirements for the project. Those requirements need to be estimated by the project managers for the deadline and allow the resources to vary.  The best projects use formal requirements analysis so that they start with complete and consistent requirements; think about it, would you ever build a house without a solid blueprint?

Formal requirements analysis will increase productivity by 16.3% and code quality by 23.2%.

If the resources exist in the organization then they need to be dedicated to the high priority project and project scope trimmed if possible and the quality of the features need to be weakened until a feasible project is defined. If trimming the scope and bringing quality to a minimum will still not result in a feasible project then you need to consider what it means for the project to be late (because it WILL be late).  Your best chance of trimming scope correctly depends on formal scope management.

Formal scope management will increase productivity by 13.5% and code quality by 18.5%.

Any project which increases the overtime by the IT department must be done on a sustainable basis. The more time you demand from the IT department the less sustainable it is, i.e. don’t expect your IT department resources to do 80 hour weeks for 6 months.  Heavy overtime will lead to friction among team members.

Friction among team members will decrease productivity by 10.0% and code quality by 15.0%.

With high priority projects make sure that the engineers get some kind of bonus on a regular basis (free lunches, dinners) as well as a financial bonus on completion of the project. If no bonuses are available don't be surprised to see IT resources jump ship at or before the project completion date.


Articles in the "Loser" series

Want to see sacred cows get tipped? Check out:
Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book at least once :-)