blogs.conchango.com

General Eisenhower said that "The plan is useless; it's the planning that's important." How I understand this quote is that the General knew that even with all the planning he participated in, some people would end up on the wrong beaches, there would be a loss of some critical supplies, and many soldiers would face life-threatening obstacles that were never imagined in the planning process. However, he knew he had created an ongoing planning process that was understood by his people that would allow them to recognize and address the most seemingly insurmountable problems. I think that if the General could successfully pull off this planning process to win a war, surely we can do this in our everyday jobs. 

I am writing this blog to explain my thoughts on why I think the Agile project planning process (Release plans) are more effective than Waterfall projects plans.  Here I make reference to PRINCE2 as a waterfall methodology as this is a process I am familiar with.  I note here that PRINCE2 is not necessarily a Waterfall methodology, rather a process for managing and planning any project, but it is often used to deliver projects with the Waterfall approach of design up front in mind.

I also wish to address three specific issues raised in the following article.

http://www.claretyconsulting.com/it/comments/agile-scrum-fails-to-get-to-grips-with-human-psychology/2006-08-17/

Namely, Agile

1. [is not based on] Commercial production decisions are based on rational expectations.
2. [that] Resource Management had vanished.
3. Each of these [Agile] organisations had differing development approaches [within the same IT department] and tools from project to project.

Anyone not familiar with Agile planning and is interested can get an excellent introduction to this topic by reading Agile Estimating and Planning by Mike Cohn.  Here I will stick to the Scrum estimating and planning process described in Agile Project Management with Scrum by Ken Schwaber (except for the use of story points and velocity which I think greatly enhance the Scrum planning process).

As with any project a Scrum project cannot start without a business case, funding and an aspirational set of requirements for the project.  In Scrum this set of requirements is called the Product Backlog which is an estimated and prioritised list of requirements that are written as goals (the what, not the how) and preferably with defined acceptance criteria.  These Product Backlog items are then grouped into four week iterations (or less) based on business value (high priority, low cost) and architectural importance that could be completed by a certain team size (which includes analysis, development, documentation and testing).  This grouping into iterations gives what is called the Initial Release Plan as it is the first estimate of what could be delivered by when.  As emphasised by Mary Poppendieck and Tom Poppendieck in Lean Software Development in the chapter on Waste, the art of gaining maximum benefit from software resources is to release it as early as possible and do it as often as possible.  Until software is being used it just costs a lot with no benefit, so I try to champion this view when forming the Initial Release plan. 

As part of this planning exercise the types and numbers of resource needs to be determined.  To do this and to support the business case a basic understanding of the technologies to be used also needs to be determined.  This will not only be driven by the business requirements, but also the IT department’s strategic requirements around, including tool selection and architectural guidelines. 

A waterfall project will also go through a similar planning exercise. In the PRINCE2 methodology this is done in two phases, Start-up Project and Initial Planning where the business case is defined or verified, a quality plan is produced and a high level plan that is broken down into a number of stages.  Again this will involve resource planning and certain decisions about the use of technology.

So far, these sound reasonably similar, but they are some key differences that should not be overlooked.

• The Scrum Release plan is broken down into fixed length iterations where a PRINCE2 Stage can be any length
• In Scrum each sprint targets producing something that is production quality, i.e. it could be picked up and implemented in the real world “tomorrow”.  PRINCE2 stages do not have stipulation and could produce artifacts that, on their own, hold no value in their completed state.
• The technical design in a Scrum project may be a lot more high level than that of a Waterfall project.  As suggested in Lean Software Development, Agile projects try to make decisions as late as possible.  This could also be the case in PRINCE2 but in many Waterfall projects the technology is often decided to quite a detailed level up front (as getting it right up front is the basis for Waterfall).  A SCRUM project may start with a choice of technologies to be used, that could all satisfy the business and technology requirements.

I think the key here is that both Waterfall and SCRUM project plans, if done properly, allow commercial production decisions to be based on rational expectations.  I hope this is enough to convince you that condition 1 is met by both methodologies at least at the start of the project.

Similarly, both involve an initial resource plan so item 2 is also addressed (at least at the start of the project).

I also think item 3 has been addressed in full, regarding organisations having differing development approaches within the same IT department and tools from project to project.  I would suggest that either there was a lack of strategic direction within the IT departments or it was felt within those departments that no form of strategic direction is required in an Agile organisation.  Personally I think this is a mistake, and a balance must always be struck between the business and technology requirements.  Ken Schwaber makes specific reference to this in Project Management with Scrum when discussing multiple Scrum projects. The first sprints are concerned with getting the technology and operational structure right for multi-team development whilst still demonstrating some business value.  In Lean Software Development in the diagram of the Agile value chain also describes the need for this initial architecture. 

The heart of a Scrum project is the Sprint.  A sprint starts with Sprint planning where the Product Owner (the person responsible for delivering business value) collaborates with the team (those tasked with delivering the Sprint) to agree which Product Backlog Items can be delivered in the Sprint.  (Note that this may be different from the Initial Release Plan due to further understanding of the requirements.)   Having agreed this, the Team identifies and estimates the tasks that are required to produce an increment of developed and tested code in the Sprint.  This list of tasks is called the Sprint Backlog.  The Sprint then starts and the team assesses the Sprint Backlog on a daily basis including revising time-to-go or creating unforeseen tasks or deleting redundant tasks.  This is communicated in the Sprint burndown chart which plots the time to go against the number of days in the Sprint.  This gives a clear indication to the team of how the team is progressing against the plan.  At the end of a Sprint the team demonstrates their increment of functionality to the major stakeholders and feedback is given. This may identify additional requirements or items where functionality that did not meet expectations or may result in the realisation that certain requirements have become redundant. The Product backlog is updated according to what was delivered which gives a new estimate to go.  It also gives an idea of the velocity the team is working at.  This is either done on a Product Burndown Chart which plots Sprints against effort remaining, and/or by calculating the velocity which I calculate as

Velocity (story points per unit man day/hour) = Story points in Sprint DIVIDED BY man hours/days dedicated to Sprint

Note, that for these purposes, the story point can be considered as the original man-time estimate made when the Product Backlog was created.  This information can be used to re-assess initial Release Plan to gain an improved estimate of what can be delivered when.  It will prompt questions such as

• can we go-live early, we better get the production hardware ready a month early?
• Oh my god we are never going to get this done in four months, maybe we should extend the go-live date now or reduce the scope of the project so that the majority of the business case is delivered on time?
• Should we change the resource profile?

The point is that the Release Plan is constantly evolving with each Sprint.  This is based on, what I believe to be the best metric in software development and that is amount of working, acceptable code delivered.  This again is a principle that supports the Agile Manifesto: Working software is the primary measure of progress.

In a waterfall project using PRINCE2 a stage plan will be drawn up during the previous phase.  Although this may be done in conjunction with the team, the project manager owns the plan.  The project manager then tracks the project against this plan getting regular updates (which can be daily) of time spent and time to go.  The project manager will try to deliver the stage within certain constraints called tolerances and if he/she fails to do so an exception plan will be created to reflect the new timeframe or cost and resource profile.  The exception plan can be for the Stage and where necessary the overall Project Plan. At the end of a phase the stage boundaries process is implemented to review overall project progress and the plan.

Again similarities can be drawn between Stages and Sprints but there are some clear differences that need to be understood:

• Sprints are of defined length (apart from in extremely exception circumstances where a sprint could be stopped) where as stages can vary in length due to an exception plan
• Sprints deliver an increment of shippable functionality whereas Stages may not.  A Stage may produce a functional requirements document or a technical design. 
• The project manager owns the Stage plan in PRINCE2 whereas the team own the Sprint Backlog

Again, if implemented properly both Scrum and PRINCE2 satisfy both item 1 (Commercial production decisions are based on rational expectations) and item 2 (Resource planning) throughout the project lifecycle.  They are both planning processes adapting to plans.

So why do I think Scrum release plans are better that Waterfall project plans?  This comes down to following factors:

• Waterfall encourages getting the plan right up front.  If a waterfall project is not going to plan, people being human, will sometimes attempt to keep things to the original plan if they feel their reputation is at stake.  This will lead to a “head in the sand” approach to delivery, where delivering to the original plan must be achieved at all costs, resulting in a lot of issues being pushed to the very last phase in the project often resulting in an unexpected delay in go-live right at the end of the project.  I think when this occurs Commercial decisions are being based on anything but rational expectations.  Conversely, if your stakeholders are groomed properly they have different expectations set and are more willing to accept hiccups and wins as they are seeing working code every four weeks in a Scrum project. I am not saying the “head in the sand” approach does not occur on Agile projects, I would say I have run two of these where through contract or stakeholder belligerence the scope is fixed as well as the timescale resulting in the very same issues as found on waterfall projects when the “head in the sand strategy” is adopted.
• In Scrum, the team own their own plan for the iteration whereas a project manager owns the plan in most (if not all) waterfall projects.  The distinction may seem arbitrary, but I am strongly in favour of the team owning the plan, even if I can feel a little in the dark sometimes about what is going on.  When I have managed both Scrum and non Scrum projects I have found that team members can give very poor estimates to go just because either they do not want to have to think about it or are scared to give bad news, often resulting in late notification of delays.  By placing ownership and responsibility for the Sprint Backlog within the team, the team tend become better time managers and are less worried about giving bad news.  I think the reasons for this is are that the team learns the value of monitoring their time to-go when they need to deliver something in four weeks and are more honest to the team where previously they would have to explain themselves to a perceived superior.  (I must confess, however, that getting the team to use the Sprint Backlog in the first place is a difficult exercise, as this is an additional responsibility that would have traditionally sat with the project manager and can be mistaken for an administration overhead by team members.)
• The Scrum release plan process targets the highest business value early, whereas Waterfall projects often have a fixed scope (a defined product in PRINCE2) so that this business focus can be lost.  Imagine a project that is going to take twice as long as originally expected.  The Agile project allows for high value code to be released on time with a second release later or with abandonment of lower priority requirements.  A Waterfall project plan does not have this flexibility and could be abandoned due to overspend without any business value being derived at all.
• The Scrum release plan process is based on a more effective metric – working code.  All elements of the delivery are being measured from the first sprint.  The analysis, development, test and documentation is all performed in a sprint.  If your going to get a surprise in Agile regarding estimates it is usually going to surface in the first two or three sprints usually well in advance of the release date.  Waterfall metrics tend to use measures derived from other projects and are therefore not attuned to the nuances of the specific project.  If functional design overruns then development is likely to overrun by the same % factor.  This may not always be the case.
• Reduction in number of non-critical artifacts.  Although PRINCE2 focuses on products and not tasks, Waterfall does imply a design up front process.  This means that documentation is created to inform future phases, but may have little use when the project is completed.  Agile favours working software over comprehensive documentation.  This does not mean that Agile projects do not produce documentation, but the documentation is minimised.  This means, from my experience anyway, that when coding from day 1 is the focus, this leads to more functionality for less money.
• Agile planning brings together the business and technology arms of a project in the same room to conduct its planning and to review the outputs regularly.  This gives both parties a feeling of control over the outcome of the project because they can debate what is wanted versus want can be delivered on a regular basis.  Waterfall projects can tend to show the business a plan up front and a set of requirements that the business is told it must (try to) stick to and show there commitment to this by signing the requirements off as a complete statement of their requirement.  In most cases divergence from the plan will occur, either through the business realising that what they asked for was not actually what they needed, or the project is just harder or easier to deliver than was expected so the project undergoes an often laborious change control process to the “contract” that was made at the beginning.  As the change control process is written in paper I find a lot of resentment builds up between parties as neither side feels they are being properly understood.  I find that face to face direct communication tends to resolve these issues but only if people are brave enough to assert themselves.