In 2007 the UK Department for Communities and Local Government (the DCLG) entered into a contract with European Air and Defence Systems (EADS, now known as Cassidian) to deliver an IT system that would underpin the FiReControl project. The FiReControl project aimed to improve the UK Fire and Rescue Service by replacing 46 local control rooms with a network of nine purpose-built regional control centres. The project was expected to be completed in October 2009, and the DCLG’s original estimate of the project costs was £120 million. By 2009, two years into the project, the expected duration of the project had doubled, and the anticipated total project costs had increased by more than 500% to £635 million. In 2010 the contract was terminated. The DCLG had wasted at least £469 million as a result of the failure of the project to deliver.[1]
This is a sobering story of a large IT project spiralling out of control. But it is not an isolated incident. Indeed, about two thirds of all software projects are delivered late or fail outright.[2] Not only that, but one in six IT projects has a cost overrun of 200% on average and a schedule overrun of almost 70%.[3] It seems that no organisation is immune from these risks. There was a common belief that out of control IT projects were the preserve of the public sector, but recent studies show that the private sector does not fare any better in comparison. Organisations in the private sector are simply less publicly accountable and so have greater ability to conceal IT disasters.
So why are IT projects so high risk, and how can this risk of failure be mitigated?
There is a huge disconnect in the world of software development. The lawyers aim to specify the relationship between the contracting parties with as much certainty as possible. But both the business function and the software development practitioners are operating in an increasingly complex, dynamic and inter-connected environment.
The legal and management functions have, by and large, not adapted their practices or values to take account of the challenges faced by the business functions and software development practitioners. Indeed, the contract model for software development (the ‘Contract Model’) and the management practices that surround it have barely changed in the last 30 years. Much of the thinking underlying the Contract Model can be traced back to principles that were developed during the Industrial Revolution by the likes of Henry Ford and Frederick Taylor.
Our view is that the Contract Model compounds the effects of poor management, and that poor management is often based on the flawed thinking underlying the Contract Model. We have found that, even if an IT project is resourced internally, the organisation tends to apply quasi-contractual relationships between its internal departments for the purchase of IT services. And we find the principles of the Contract Model in evidence here too.
We do not believe that we will see any significant improvement in the success of IT projects until we change the basis of the Contract Model and the surrounding management practices. For the purposes of this article we use the term ‘IT project’ as shorthand for any IT change initiative involving the development of software.
Fundamentals of the Contract Model
We believe that any contract for software development based on the Contract Model contains three distinguishing features, all of which are seriously flawed. We use the terms ‘supplier’ and ‘customer’ to explain the dynamics in an external relationship. However, as mentioned above, in many cases similar principles appear to apply even if the IT project is resourced internally.
The three distinguishing features of any contract for software development based on the Contract Model are as follows:
· Output-based Requirements. The supplier is required to deliver output that possesses all of the requirements, as specified by the customer in the contract. We use the term ‘output’ to refer to product (eg code, features, functions, attributes), documentation and/or services.
· Change Control Mechanism. The Contract Model mandates that any change to any Output-based Requirement or to any other element of the contract is regulated by the Change Control Mechanism. Broadly speaking, to initiate change the customer must submit a change request form to the supplier outlining the desired change. The supplier analyses the impact of the change request on the contract as a whole, including, in particular, the Output-based Requirements, the price and the due date for completion of the IT project. On the basis of this, the supplier proposes an amendment to the contract. Following the agreement of the parties to the proposed contract amendment, a formal amendment is made to the contract to incorporate the requested change.
· Sequential Development. The software is to be developed sequentially, ie using the waterfall model. Development is seen as flowing steadily downwards through the phases of conception, initiation, analysis, design, construction and testing. The supplier is required to complete each phase before starting the next phase, and the output of each phase provides the input for the next phase.
It is worth noting that we do not make any reference to the type of charging model in the Contract Model. If the three distinguishing features of the Contract Model feature in a contract, the contract will be flawed regardless of which charging mechanism is adopted.[4] It makes little difference whether the price is fixed, target or based on time and materials, or whether there are bonuses or penalties.
On the face of it, and to anyone not directly involved in the implementation of the IT project, the Contract Model may appear to be eminently sensible. It creates a sense of certainty and predictability regarding the IT project, and it provides a clear and understandable structure for the various activities involved in the IT project. Plus it mirrors current procurement models.
However, the combination of these three elements in a contractual relationship between a customer and a supplier appears to cause the failure of many IT projects that would probably otherwise succeed. In fact, we would suggest that those IT projects which do achieve success do so in spite of the Contract Model and the surrounding management practices, and not because of them.
In this article we explore some of the ways in which the Contract Model increases the risk of failure in any IT project.
An increased risk of failure
Any IT project is subject to risk which can be categorised into three main types:
· Delivery risk. This is the risk that the IT project is not delivered on time, on budget and to the required quality.
· Business value risk. This is the risk that the IT project doesn’t deliver the expected business value.
· Existing business model failure risk. This is the risk that the IT project damages the existing organisation.
The Contract Model does not address the second two categories of risk and actually increases the customer’s exposure to all three categories of risk.
Delivery risk
The FiReControl Project is a classic example of the delivery risk derailing the IT project. The UK National Audit Office noted that during the first two years of the contract there was little progress in delivering the IT system.[5] Indeed, the DCLG does not appear to have received any working software before the contract was terminated. We do not know the reason for this but we can hazard a guess.
The outdated principles of the Contract Model assume that the Output-based Requirements can be predicted upfront. As a result the Contract Model fails to respond adequately to changing conditions and forces the customer to incur costs disproportionate to the returns on the IT project.
The fallacy of predicting the Output-based Requirements upfront
The assumption that the Output-based Requirements can be predicted upfront is predicated on two further assumptions. Firstly, that the Output-based Requirements are understood fully by both the customer and the supplier and, secondly, that the software can be finished before significant changes occur. In other words, to function optimally, the Contract Model requires both contracting parties to possess perfect information. This is a practical impossibility.
The very dynamics of an IT project lead to changes. It is only natural that, as the customer learns more about the latest technology and its relative strengths and weaknesses, the customer revises its thinking on how best to take advantage of the technology.
‘… [S]ystems requirements cannot ever be stated fully in advance, not even in principle, because the user doesn’t know them in advance – not even in principle. To assert otherwise is to ignore the fact that the development process itself changes the user’s perceptions of what is possible, increases his or her insights into the application’s environment, and indeed often changes that environment itself.’ [6]
By the same token, it is only natural that as the supplier learns more about the customer’s business processes, it will gain a deeper understanding of the problems that the customer is trying to solve and the opportunities that the customer is trying to harness.
In any event, it is not possible to anticipate how the many disparate design elements of the software will interact before the software is implemented, and this can frequently lead to surprises.
The assumption that the software can be finished before significant changes occur is also flawed. External forces are at play. Technology is evolving at an ever increasing pace. The market or context in which the concept for the software was conceived continues to change. Hence, the opportunities or risks to be addressed by the software also change. For example, the emergence of disruptive technologies such as Facebook, Twitter or the touch screen iPad can have a huge impact on any existing plans for software development and distribution. The regulatory environment may also change.
Recent studies, led by Al Goerner at the University of Missouri, Kansas City, demonstrate that the inherent value in Output-based Requirements erodes exponentially over time. This rate of decay has been likened to the half-life of an unstable radioactive atom. The ‘half-life’ is the measure of the period of time it takes for the substance undergoing decay to decrease by half.
According to the studies carried out by the University of Missouri, the half-life of the value of the Output-based Requirements has been rapidly decreasing. In 1980 this was around 10-12 years, by 2000 it had fallen to 2-3 years, and it is currently running at about 6 months.[7] In other words, half of the Output-based Requirements will become obsolete by the end of month 6, half of the remaining half (i.e. 1/4) will become obsolete by the end of month 12, half of the remaining quarter (i.e. 1/8) will become obsolete by the end of month 18, and so on. Hence, by the end of month 18, according to the University of Missouri studies, only 1/8 (i.e. 12.5%) of the Output-based Requirements will still possess any inherent value.
In its desire to create certainty, the Contract Model actually creates the risk that, when delivery finally occurs, what is delivered may no longer meet the customer’s needs.
The failure of the Contract Model to respond adequately to change
If changes happen, the customer will wish to change the Output-based Requirements. As an integral part of the contract, the Output-based Requirements cannot be amended to reflect a change without a formal amendment to the contract as agreed by the parties in accordance with the Change Control Mechanism. In a fixed price contract, the amendments are often seen as leading to additional work and the customer is required to pay additional fees. It is not uncommon for the supplier to view a change control as an opportunity to increase its margins.
The initial stage of the Change Control Mechanism is that the supplier analyses the impact of the change requested by the customer. The larger and more complex the IT project, and the greater the amount of work that is involved in the supplier carrying out this exercise. Sometimes, the impact of the change request is so complex that the supplier simply cannot work out how to incorporate the requested change into the existing IT project.
The process of analysing the impact of a change request can take so long and be so extensive that it has a destabilising effect on the IT project. The main team is only too aware that current work may be rendered redundant following the approval of the change request. The bigger the proposed change, the longer the hiatus while it is being analysed, and the more damaging the effects can be.[8]
Any change request will inevitably cause delay to the IT project. It is unlikely that the original timetable builds in a buffer for the supplier’s resources to be diverted to this activity and for any resulting additional work to be undertaken. It is for this reason that both customers and suppliers consistently cite changes to the Output-based Requirements as one of the main causes of failure of an IT project.
To make matters worse, the Contract Model mandates Sequential Development. It is not until testing, late on in the IT project, that the customer has visibility of the software. Until then it is very difficult for anyone to assess whether the IT project is on track. The deliverables of all earlier phases are documents that are based on assumptions. It is only when the software is actually built that anyone can accurately assess whether the IT project is on course to meet the Output-based Requirements. However, there is a long gap, often in the order of years, between the date when the customer collects the Output-based Requirements and the date when the supplier makes the first delivery of working software. The longer this gap, the more likely that significant change has occurred during the intervening period.
Business value risk
The FiReControl project highlights the importance of demonstrating from the outset how the ICT project will deliver the expected business value and of obtaining the buy-in of all those involved. The DCLG was criticised by the National Audit Office for not making sufficiently clear the case for a centrally-dictated standard model of emergency call handling and mobilisation, operating from new purpose-built regional control centres. From the start many local Fire and Rescue Authorities and their Fire and Rescue Services criticised the lack of clarity on how a regional approach would increase efficiency.[9] Unless the resulting software delivers tangible business value, it doesn’t matter how state of the art or sophisticated it is, the intended end-users may each simply decide not to use it.
The business value risk, overlooked by the Contract Model, is far more serious. There is, instead, an assumption that, if the supplier delivers software that meets the Output-based Requirements, it will therefore deliver business value to the customer. However, this in turn assumes that the customer knows what it needs. What we have found instead is that, although customers are very good at stating what they want, far too often customers do not in fact know what they need. As a result, it is not uncommon for the customer to be disappointed with the resulting software, even if the supplier can demonstrate that the software meets the Output-based Requirements.
It is a sad indictment of the current state of software development that one of the greatest risks is that the supplier builds ‘the wrong product’. This happens whenever the supplier successfully executes against the customer’s specified Output-based Requirements, but the resulting software does not add any real value to the customer. The software does not add value because it does not enable the customer to solve the problem that it had wanted to address.
This is best illustrated by the findings from the US Department of Defense (the DoD).[10] The DoD analysed the results of its software spending, totalling an eye-watering $35.7 billion, during 1995. They found that only 2% of the software was able to be used as delivered. The vast majority, 75%, was either never used or was cancelled prior to delivery. The remaining 23% was used only following modification. That would suggest that the DoD actually received business value from just $0.75 billion of its expenditure – nearly $35 billion of its expenditure did not result in software that delivered any immediate business value.
The reason why customers to date have derived so little business value from the software delivered by the supplier is that the Contract Model is not referenced to the target outcomes of the customer (ie the results that the customer wishes to achieve and which will add value to the customer). Instead, the Contract Model is referenced to the Output-based Requirements, ie the requirements for the deliverables of the IT project which are intended to contribute to and facilitate the achievement of the target outcomes.
People buy a hammer to knock in a nail so that they can put up a picture – they know that they can achieve their target outcome (putting up the picture) with the acquisition of the hammer. Unfortunately, in the context of software development it is not as straightforward to make the link between the delivery of the output (the software) and the achievement of the customer’s target outcomes. Many people simply don’t even try. This creates a large risk that the supplier will only deliver what the customer asked for – a vague set of Output-based Requirements – rather than what the customer actually needs, which is to achieve the target outcomes.
Software development involves the transformation of ideas into deliverables to achieve business objectives. The catalyst for the IT project is generally a business case. This justifies, at a strategic and financial level, the acquisition of the software. The anticipated cost of the software is justified by various assumptions such as the improvement of business processes, increase in market share, increased revenue, reduction of support costs and so on. Following internal approval of the business case, the Output-based Requirements are then collected and assimilated from everyone at the customer’s organisation who has an interest in the resulting software system.
So if a business case generally precedes the specification of the Output-based Requirements, why is it the case that the resulting software will not necessarily meet the target outcomes?
Firstly, the business case is untested. In many business cases there are elements which are based on assumptions rather than facts. Many of them may be erroneous. Ideally, those assumptions should be tested before significant resources are invested in building a software system that delivers against the business case. But that rarely happens.
Secondly, the business case is typically produced at a high level and with a view to obtaining funding or budgetary approval. It is not uncommon for the business case to be very ambitious in terms of what the software will achieve, as this provides a better argument for investing in the acquisition of the software. It is less common for the business case to play an active role in steering the direction of the IT project. It is even less common for anyone to revisit the business case in the light of the progress of the IT project or to measure the progress of the IT project by reference to the business case.
The implications of this cannot be overstated. The DoD is one of the most sophisticated IT purchasers worldwide: it has a significant amount of leverage in negotiating contract terms because its annual spend is so large. And yet it derives practically no immediate business value from its investment in software development. Is it possible that other organisations, with less experienced procurement functions, actually derive less business value from their IT spend?
The most effective way for any organisation to reduce its IT spend is to ensure that only software which delivers business value is built. We need to consciously connect the levels and clarify how the resulting software will deliver business results.
Existing business model failure
The Contract Model does not address the possibility of existing business model failure risk. There is simply no recognition of the fact that, when a new software system is launched, this may impact on the existing business processes. As a result, the Contract Model increases the exposure of the customer to existing business model failure risk, which is arguably the most insidious of the three categories of risk.
Perhaps, back in the 1980s when the Contract Model was first used, software systems were fairly discrete and limited in terms of their operation. However, today software systems are used for virtually every business function of an organisation. There may be end-users at multiple levels of the organisation – eg, the finance director, accounts department, marketing director, marketing team and sales team may all need to use the same software system. This software system may interlink with other software systems within the organisation which are used by other end-users. The software system may also interface with software systems of other organisations – such as those of clients or suppliers.
In light of the many business processes that may be impacted by a new software system, it is essential that the transition to this system is managed in a way that reduces the risk of existing business model failure to a minimum. However, the Contract Model generally requires that all of the Output-based Requirements are delivered as a single batch. The larger the IT project and the larger this batch will probably be.
For many organisations it is simply not realistic to attempt to assimilate a software system of this scale and complexity into their existing business processes all at once. The risk of any of those existing business processes falling down under the enormity of the change is huge. It would be much better if the transition to the new system was managed in smaller launches, with an emphasis on the quality of the user experience throughout the transition.
Conclusion
Much research and many studies have been carried out on IT projects to try to understand why there are so many failures and why the extent of those failures can be so great. Yet the Contract Model has been largely ignored. We believe that the Contract Model is in need of a total overhaul. With our increasing dependency on IT and escalating costs of IT spend, an overhaul of the Contract Model cannot happen soon enough.
Susan Atkinson is a Consultant Solicitor at Keystone Law and Gabrielle Benefield is a co-founder of the Scrum Foundation and founder of Evolve Beyond.
[1] ‘The failure of the FiReControl project’ Report by the Comptroller and Auditor General of the National Audit Office, 1 July 2011.
[2] ‘The CHAOS Report’ by The Standish Group, 2011. Whilst the methodology and the conclusions of the CHAOS Reports have been called into question, these reports still remain the most comprehensive surveys to date, and in our experience anecdotal evidence is largely consistent with the findings of the CHAOS Reports.
[3]‘Double whammy – How ICT projects are fooled by randomness and screwed by political intent’ by Alexander Budzier and Bent Flyvbjerg, Said Business School working papers, Oxford, August 2011.
[4] Sometimes the contract does not contain sequential development but, even if the contract only contains the Output-based Requirements and the Change Control Mechanism, it is almost as problematic.
[5] Infra footnote 1.
[6] ‘Lifecycle Concept Considered Harmful’ by Daniel McCracken and Michael Jackson, ACM Software Engineering Notes, April 1982.
[7] We have been unable to find out more details regarding these studies. Clearly the half-life for the specifications will vary for different sectors. However, there is anecdotal evidence to support the view that the half-life could be even shorter in the technology sector.
[8] ‘The curse of the change control mechanism’ by Susan Atkinson and Gabrielle Benefield, Computers & Law, May 2011.
[9] Infra. Footnote 1.
[10] The results of the study were presented at the 5th Annual Joint Aerospace Weapons Systems Support, Sensors, and Simulation Symposium (JAWS S3) in 1999.
