Waterfall is for Losers?

Perhaps that is a bit harsh, but in the question of whether Agile or Waterfall is best, the reality is that Agile ends up being multiple small waterfall cycles.  So you end up doing Waterfall in pieces.

All projects are composed of a set of tasks, typically executed by multiple people.  Within your set of tasks, some are dependent on others.  In particular we can never get away from the fact that to implement any single requirement, we need to analyse the requirement (what), design the solution (how), code the solution (execute), and test the result.

So any iterative and incremental project, i.e. Agile, Scrum, RAD, XP, etc.  Is really a series of small waterfalls.  The difference is that between the waterfalls we have sandwiched requirements, analysis, design, and coding which gives us the ability to change direction when we need to.

Note: If you never need to change direction (which is not often :-)) then Waterfall is a viable option.  Waterfall makes the most sense in projects where there is little to no requirements or technical uncertainty^*, e.g. reimplementing a system with experienced developers in a new technology where the use cases for the previous system are not changing.

The typical waterfall project assumes the above picture, that all requirements can be done then all design all coding and then all testing.  It is based on a 1960's factory mentality that code can be assembled like a car on an assembly line.

Waterfall only works if the amount of rework in any phase is immaterial and does not materially affect the length of the project.  Typically this is not true.  In the presence of requirements uncertainty, requirements need to be revisited many times.  Often, missing or inconsistent requirements will cause scope to change requiring change requests in the process.

All change requests will affect the project schedule.

In the presence of technical uncertainty, design needs to be revisited many times.  Often when using newer APIs the API is not documented or even worse, the code mechanism from the creator/vendor simply does not work and requires technical work arounds that materially affect the project schedule.

This will create a change request that will affect the project schedule.

An agile project builds requirements and design into every sprint.  This allows you to change directions when either requirements or technical decisions are uncertain.

This is the series of small waterfalls -- you are still doing Waterfall, but in an Agile way.

When you add up all the sprints of a project, you will realize that you have done the same project (i.e. the work is the work), but you have sliced it up differently.

So it is not a question of whether you will be doing Waterfall or not.  It is a matter of what is the sprint cycle that will support the requirements and technical uncertainty of your project.  If your project has little to no uncertainty, then your sprint cycle is the entire project.

If your project has requirements or technical uncertainty, then sprint cycles of 2-3 weeks are ideal.

^*Note: Similar to requirements and technical uncertainty are projects where there are significant dependencies on client interaction, so schedule uncertainty can also drive the need for agile project management.

---

Other articles in the "Loser" series

Want to see more sacred cows get tipped? Check out

• Comments are for Losers
• Defects are for Losers
• Debuggers are for Losers
• Efficiency is for Losers
• Testing departments are for Losers

Make no mistake, I am the biggest "Loser" of them all.  I believe that I have made every mistake in the book once :-)

What is software architecture?

What is software architecture? In the previous article "What is architecture?" we stated that architecture is about:

• Structural elements
• Connective elements

Let us talk about these with regard to a software system.  In our next article we'll talk about good and bad architecture.

Structural elements

Structural elements in software are layers that support other layers in the system:

• the language libraries are built on the functionality provided by the O/S
• the functionality of the software product is built on the language libraries
• 3rd part libraries that you are using are built on the previous 2 layers
• your code is built on all the previous layers

So any program you build has at least the following structural elements:

In a well designed system, your code will also break into layers specific to the application that you are building.  Each layer ideally only depends on layers underneath it, otherwise you will have circular references; asignal that there is a challenge or that something is wrong. 

Connective elements

Every software system has connective elements that communicate across the different layers.  Often 3rd party libraries are shared services that provide core services like data structures (XML, JSON, etc), logging, debugging, and other services.

Inside your code you will also have shared services that are used by every layer, this is diagrammed as follows:

Where layer 1, 2, ..., N represent the layers in your code and the share services are shown vertically as they cut across all the layers.  Layer N represents the UI layer of the system and the functionality at that layer depends on all the layers below it.

Generalizing Architecture

When people are talking about software architecture, they are not just talking about the structure of the code.  Often people are talking about the machines and O/S components as well.  So for example when people are talking about the LAMP architecture they are talking about:

• The O/S -- i.e. Linux
• The web server -- Apache, running on Linux
• The database -- MySql, running on Linux,
• The code -- PHP running on the MySql database using services from the O/S

Note: The PHP layer is further broken down like the diagram above in the Structural Elements section.

The LAMP architecture involves structural elements.  Connective elements are simply the ones created inside the PHP layer or generally provided by the O/S.

So when relevant, architecture includes:

• The O/S
• All support services, i.e. webserver, database, etc.
• The high level applications written by you or a 3rd party

General Software Architecture

For the structural elements of software we are taking about layers.  Those layers can be relative to the O/S such as how the O/S communicates with support services (web servers, databases, etc) or how your applications communicate with support services and the O/S.

Within your application there are different layer interactions, there is 1) how your base layers are built on the APIs of your language and the 3rd party libraries that you have installed, and 2) how you have broken up your layers to communicate with the different parts of your software.

The main structural elements of any model-view-controller (MVC) program looks somewhat like this:

• The model layer a.k.a. business layer is built on 3rd party APIs and O/S APIs
• The view layer a.k.a. GUI layer is built on 3rd party GUI APIs
• The data layer is build on database APIs

All of the APIs that you are using are based on the standard APIs of the language that you are using.

Implications

• Changing languages requires that every layer above it be reconsidered
• This means that if you change languages and some 3rd party API must be changed then the layer above also needs to be changed
• Strict separation in layers means that you can change a 3rd party API and only a single layer of the MVC system is affected
• Often developers color between the lines and a change of a 3rd party library causes quite a bit of pain as the layer infractions are found
• Not having enough of a business layer will be problematic because it is one of the main separating structures between your view/GUI and data layers.
• People discover lacking business layers when they change their database APIs or databases and have the problems propagate all the way through the GUI
• Often when business functionality is built into the model/GUI layer then that functionality is lost and needs to be recoded when a new GUI API is used

Connective elements

Connective software elements are used to shuttle data between the structural layers of the system. These connective elements are generally available everywhere (although they can be limited to a few layers).

Easily understood connective elements are those used in debugging and logging.  These elements are available universally, e.g. debugging and logging routines can be made available in all layers without worrying about layer dependency.

Other connective elements are most easily recognized because they are routines shared across layers of the architecture.  More specifically, they are the libraries/modules that are included into multiple layers of the program.

---

So now we have defined what structural and connective elements are in software, next we describe what it means to have good architecture and bad architecture.

What is architecture?

On a regular basis we hear people talk of good and bad architecture, but what is architecture?

Before I describe software architecture, let's see if we can come to an agreement of what architecture is.  What are the components of architecture, and what value does architecture have.

Architecture provides the structural and connective framework required for a system of components to function.  Architecture is specific to a context, good architecture for a software system is different from that of a car or a building.

Architecture in general is not visible, it is present in the system but under other visible design components.  Before a discussion of software architecture, it would be best to describe architecture using physical objects.

Architectural Elements

Architecture involves two elements:

1. Structure
2. connective elements.  

Let's look at these two elements with respect to a building.

Structure for a building involves the foundation and the pieces that provide support to the entire building; it is the skeleton of the building.  If a building has an interesting shape it is because underneath the framework of rebar and concrete support the shape.

Connective elements can be structural, but they provide a way of linking different structural components for the purposes of transporting something.  Connective elements in a building transport air, water, and electricity.

The structural capability of the framework will dictate how high a building can go; generally speaking architecture determines size.  If the building has a framework of rebar and concrete that will support a 10 story structure, it will be difficult to add additional floors over 10 easily. Adding additional floors will require effort expended to reinforce the existing structural strength of the building

If a connective element is missing then adding it will be expensive.  For example, old brick buildings often didn't plan for plumbing or electricity.  If this element is added afterwards, then it will be much more expensive to put in.

If plumbing is added afterwards then you will see the plumbing running outside the walls. This can lead to problems if the building experiences sub-zero weather. 

It is similarly inconvenient to add electricity or air-conditioning to a building which has not had these connective elements designed into the building when it was built.

For comparison purposes, let's look at architecture in a couple of  contexts:

Object	Structural component	Connective elements
Building	Steel frame Concrete Bricks	Plumbing Electrical system HVAC system Elevator Emergency stairwell
Car	Chasis Tires	Drive train Steering column Gas conduits Electrical system HVAC system

Architecture and Visibility

In the two examples above, several things about architecture stand out:

1. Structural components are generally hidden unless they are functional
2. Connective elements are covered up in the final object

So for a building the structural element of the steel frame is invisible hidden under finishing elements. But there are cases where we see concrete or brick walls exposed.  Structural elements are generally not attractive and so we put in extra effort (i.e. cost) to hide the structural elements. Sometimes in the case of concrete or brick walls we will leave them exposed because the visual need to hide these elements are not there, i.e. a warehouse.

Connective elements are almost always hidden.  The sight of electrical wires, plumbing, or HVAC tubes is not aesthetically pleasing and we generally hide these elements.  If we are hiding a connective element, they are cheapest to put in when an object is being created the first time.

Cost of Fixing Hidden Connective Elements

Repairing a hidden connective element is expensive.  For example, fixing plumbing and electrical wires in a house are expensive depending on how hard it is to access the connective element.

Adding a connective element after the fact is much more expensive and much less attractive.  There are brick buildings that were built prior to indoor plumbing and electricity being available.  A good example of these buildings are the residences at Harvard or old brick warehouses that are now office space.  In the case of the Harvard residences the plumbing runs outside the residences and due to the winter weather in Boston, is subject to freezing.  Adding electricity to a brick warehouse involves running metal conduits inside the walls; since they are exposed, they are subject to water accidents.

Visible Structural Elements

In the case of a car the tires are a structural element, but they are exposed to view.  This is why effort goes into making the tires as attractive as possible, i.e. white wall tires, decorative hub caps, etc.

Purpose of Architecture

Once the architecture is set it determines two things:

1. The size of an object
2. The functional capabilities of the object

As previously mentioned, the structural architecture of a building will dictate its maximum size and the connective elements will outline its capabilities.  Connective elements are always about functional capabilities.

The purpose of structural architecture is to partition an object into sub-components that are independent and can be designed separately.  For example, in a building, the structural architecture allows you to subsequently design each of the apartments separately without worrying about how the design of one room affects another.

For a car, the main structural element of the chassis allows you to design the following sub-components separately:

• engine
• doors
• lights
• seats

By allowing sub-components to be designed separately we subdivide a problem (i.e. divide-and-conquer), which reduces the complexity of overall design.  It allows separate teams to work on the sub-components.  Good architecture facilitates strong polymorphism in the sub-components.

In an apartment, each apartment can be designed differently and by different people.  In a car, the engine can be designed by one group of people different from that designing the doors, lights, or other part of the car.

The connective components in each object either: 1) provides a shared service that is accessible to multiple components, or 2) provides a coordination of sub-components to achieve a higher level function.

Electricity, plumbing, and HVAC are all examples of shared services that are available to an entire building.  The usage of electricity in one room does not dictate the usage of electricity in another room, however, aggregate usage of electricity is sized by the architecture.

Elevators for a building and steering columns and the gas system of a car are examples of components that are coordinated across multiple levels of the object.  They provide an overall functionality when all of the sub-components coordinated are functioning.

In general, if one of the sub-components is not functioning then the entire set of coordinating objects will fail to function.  For a car, if any component in the diagram fails then the engine itself will fail.

Summary

In any object composed of multiple sub-objects there is architecture.  Architecture provides two different but related functions that is:

1. Structural components
2. Connective elements

The architecture of an object dictates how large it can be and determines how separately each sub-component can be designed.

This article focused on physical objects, the next article will focus on software architecture.

Why Outsourcing Fails

Every year, we see corporations outsource operations only to pull back back some, if not all, of the operations. When this happens, the cost of pulling back operations or splitting things over multiple countries can leave you with significant operational issues and similar or worse costs.

Successful corporations understand the need to control costs in operations. The "spare no expense" philosophy of John Hammond in Jurassic Park is a one way ticket to disaster.

But, a focus on cost reduction is what leads corporations to face-fault when they outsource.

Corporations are seduced by the idea of potentially cutting costs dramatically; why pay expensive resources to do repetitive things when overseas people can do the same work for pennies?

The cardinal rule when outsourcing is:

Only operations that you completely understand and control can be outsourced 

To understand operations is to understand the costs of exceptions as well as how they are processed and escalated. Local resources deal efficiently with exceptions because they have informal relationships in the organization and use them to get things done efficiently; those relationships are not available to outsourced resources.

When operations gets outsourced, not only do foreign employees not understand how to deal with exceptions but also:

• They are often in another time zone 
• They don't have full access to local resources 
• They don't know who to contact inside the corporation 
• They don't understand local business rules 
• The network configuration can make it impossible to communicate properly 

Understanding an operational function means that you capture all details, including exceptions, in your information systems. Often, legacy information systems capture exception information as unstructured fields (i.e. memo fields) and off shore resources will not understand this.

99% of your operations might be straight forward, but the 1% of exceptions can at best lead to longer implementation times.  Best case scenario is that you will look bad to your customers and at worst, it will kill your bottom line.

If your information systems contain exception information in a structured way then at least you can eventually train foreign resources to resolve them. But unless exception information is captured so you can control it (i.e. not in general text fields) then it is the same thing as not having it.

When outsourced resources are under strict performance guidelines, they will do anything to clear their work queues. This often involves kicking back work they perceive as an exception as improperly specified; they will claim 'garbage in, garbage out'.

This will result in miscommunications and longer lead times to implement your services. Not only will delivery times, and errors go up, and your customers will get mad.

By the time you have fired the local employees, all the knowledge of the function has left the building and you are left with a broken system.

Key questions to get answers to:

• What does each exception in operations cost? 
• How often do they happen? 
• Can the information systems record exceptions in a way that you can track and control them? 
• Are the mechanisms that local resources use to resolve exceptions be used by outsourced resources? 

If you don't have a clear answer to each of these questions then outsourcing is likely to be a disaster. When you have a clear answer to the 3 questions and understand how you will address each exception then you will have much more success outsourcing.