Reference Class Forecasting and Realistic Tech Contracts

 In this article I revisit the
pervasive problem of time and cost overruns in ICT projects – but from a
different perspective.

I look at the work of leading scientists,
economists and management consultants in trying to understand why these areas remain
problematic and establish what can be done to ensure better outcomes. In
particular, why is it that project planners routinely fail to produce realistic
estimates of how long a project will take and how much it will cost even though
they know, or should know, that similar projects have overrun in the past?

I look at reference class
forecasting (RCF) and whether asking planners to compare their working
estimates to similar past projects is likely to improve outcomes.

Finally, I consider the potential challenges
associated with applying RCF in practice together with some of the related legal
issues that could arise in the context of ICT projects.

I am a lawyer (who knows his
limits) and not a scientist, economist or management consultant. Considerable
reliance has therefore been placed on the published work of leading scholars such
as Professor Bent Flyvbjerg of Oxford University’s Saïd Business School and his colleague, Alexander
Budzier. This description of RCF and how it works in practice is fairly
rudimentary but I hope that it is true to the underlying theory and serves as a
catalyst for discussion on whether RCF should be adopted by the ICT industry.

ICT projects – overruns and disputes

One does not need to look too far
to find compelling evidence that time and cost overruns remain highly
problematic in ICT projects.

Take, for example, the
often cited study by Flyvbjerg and Budzier which is summarised in the September
2011 issue of the Harvard Business Review.
Flyvbjerg and Budzier looked at 1,471 IT projects from around the world, comparing
budgets and estimated performance benefits with actual costs and results. They
found that, whilst the average cost overrun was 27%, one in six projects was a ‘Black
Swan’ with a cost overrun of 200% and scheduling overrun (or delay) of 70%.

In a separate article on ‘Megaprojects’[1] Flyvbjerg
gave several examples of high-profile projects with substantial cost overruns,
including the NHS IT system where the overrun was a staggering 400-700%.

And ICT projects are not alone. Cost
overruns for the Channel Tunnel and Sydney Opera House (going way back) were
80% and 1400% respectively.

Other work undertaken by Flyvbjerg
(in relation to construction projects) found that, on average, a one-year delay
correlates with an increase in percentage cost overruns of 4.64%. Thus a one-year
delay on London’s $26bn cross-rail project would result in cost overruns of
$3.3m per day.[2]

In a more recent study, by the Queen Mary
School of International Arbitration in London and Pinsent Masons,[3] 49% of
respondents said they had encountered ‘IT systems
development/implementation/integration’ or ‘outsourcing’ disputes in the past five
years. And 90% of respondents said that ‘delay’ was a common reason for ‘IT
systems disputes.’

Not all such projects are
problematic and fail. Many do achieve technological success and make it across
the finish line – but often at a significantly higher cost, later than
originally planned and in a form different to that originally agreed.

Why is this and how do we explain
the tendency of planners to routinely underestimate costs and over-estimate benefits
when planning ICT projects? Why do projects get the go-ahead when it is (or at
least should be) obvious that they are bound to overrun?

Cognitive pre-disposition and the inside view

According to a number of well-respected
scholars (including Flyvberg) one explanation is that key participants involved
in the planning of such projects are deluded (but genuinely mistaken) when they
prepare their estimates. According to the experts, this deluded state of mind (or
optimism bias) is caused by a cognitive predisposition found in most people to
judge future events in a more positive light than is warranted by actual
experience.  This can give rise to a
tendency to underestimate the time and cost to complete a project, even where
people may know that similar projects have overrun in the past.

Indeed optimism bias pervades our
everyday life and affects around 80% of the population. Thus 80% of us will
likely predict deriving greater pleasure from a holiday than we subsequently do
and although divorce rates in the Western World are nearing 50% most newlyweds
estimate their own likelihood of divorce as negligible. Even experts show high
levels of optimism bias: financial analysts expect improbably high profits and
doctors overestimate the effectiveness of their treatment. Indeed the way in
which our brains code positive information (it is good at that) and actual errors
in prediction (not so good at that) means that we are programmed this way.[4] On the
plus side it has also been shown that optimistic people live longer.

Where optimism bias occurs, project
planners commonly adopt an ‘inside view’ when preparing estimates for a new
project –  a view based on their own past
experiences and the task in front of them. They do not pay sufficient attention
to actual project outcomes or the ‘outside view’ which (according to Flyvberg
and others) is more likely to produce realistic estimates and outcomes.

In an ICT context this could apply
to a planner working for a customer or someone on the bid team of a vendor.

Strategic manipulators

However, some mistakes can be
deliberate (ie rather than genuine or honest) and involve deception or strategic
manipulation (in the sense those terms are used by economists and management
consultants). This behaviour can arise when planners deliberately and in a
strategic manner understate cost estimates and exaggerate benefits in order to
obtain approval and funding for a project or, on the contractor/supplier side,
to (try to) win the contract.

Such behaviour of individuals is most
likely due to political and organisational pressures placed on them in their
roles as planners or bid managers.  Where
such individuals are also ‘project champions’ and involved throughout they may
find it hard to turn back or re-set the project even when it runs into
difficulties and they know it is wrong to carry on.

The outside view and reference class forecasting

One method of overcoming these challenges
is RCF.

RCF compares the planned project
with a reference class of similar and completed projects. Adjustments to
estimates can be made depending on the outcome of the analysis undertaken. RCF
relies on hard, statistical data rather than the planners own, ground-up assessment
(or inside view) of how the project should be estimated. The detail of the
proposed project is largely ignored. According to Flyvbjerg:[5]

‘When both forecasting methods [inside view and outside view] are applied with equal skill, the outside
view is much more likely to produce a realistic estimate…That is because the
outside view bypasses cognitive and political biases such as optimism bias and
strategic manipulation and cuts directly to outcomes.’

Here is a simple illustration:

Company C wants a new IT system. Based on anecdotal information C
knows that the type of system they have in mind is commonplace but that getting
the solution right for individual businesses can be a challenge. Projects can easily
overrun. As part of the planning process C decides to undertake RCF.

C engages a firm of consultants to gather data on similar past projects
and prepare the reference class.[6] The consultants gather
the data from their own ad hoc database and other, publicly available,
information on similar completed projects.

Armed with this information C then establishes a probability
distribution for the selected reference class. This is based on data from ten
past projects including actual time and cost to complete.[7]

Finally, C places the current project at an appropriate point in
the reference class distribution so that any probable shortfalls can be identified
and related adjustments made to existing estimates.

For example, if the proposed project is an average project (when
considered against the reference class), C can expect the final outcomes
including cost increases to be in accordance with those experienced by other
average projects.

Put another way, the proposed project is bench-marked against past
completed projects of a similar type to check whether the planner’s own ‘inside
view’ is accurate or not. Adjustments are then made if needed. This could be in the form of an
optimism bias budget uplift.

Indeed that is exactly what the UK
Department of Transport did back in 2004 (with help from Flyvberg).[8]

RCF: not a new concept

RCF is not a new concept. It has
its origins in theories developed by Daniel Kahneman and Amos Tversky which, in
due course, helped them win the Nobel prize for economics in 2002.

As we have just seen, RCF was
adopted by the UK Department of Transport back in 2004 and, according to Flyvbjerg,
similar steps were taken by the Swiss Association of Road and Transportation
Experts and the Danish Ministry for Transport – in 2006 and 2008 respectively.[9]

However, despite these (and other)
examples of where RCF has been deployed the overall take-up appears fairly
modest – certainly based on research done for the purposes of this article.  This may reflect some of the potential
challenges involved in applying RCF in practice.

Potential challenges

Where do willing parties get the RCF data from?

This appears to be a key challenge.
The theory is sound but how can willing parties routinely get hold of the data
they need to formulate an ‘outside view?’ Not everyone will have access to the
information they need for this purpose.

In the ICT area, governments and
large corporate organisations that have undertaken multiple projects may have
data sets they can use – provided the data is well-organised (by project type,
location, size, time taken and actual cost incurred) and accessible.

That said, even these larger
organisations may struggle to find ten or more similar projects (ie a valid
reference class) each and every time they want to undertake RCF.

Others with useful data sets might
include Flyvbjerg and Budzier themselves. Their study of 1,471 IT projects
sounds like a very useful starting point – plus any subsequent data they may
have gathered since that study was undertaken.

Organisations such as Gartner may
also hold useful information – so too Standish Group and other international
organisations and consultancies who, from time to time, gather and/or publish
statistics relating to the ICT industry. However, smaller organisations may not
have enough or any data to rely on.

Master data set?

Notwithstanding the probable existence
of ad hoc data sets (such as those just described), there does not appear to be
a global or master data set currently available to the ICT industry. In other
words, a giant data set that all parties wishing to undertake RCF could access
and utilise.

Is the creation of a shared database
worthy of further consideration given the potential benefits for the industry
as a whole?  And could this provide an
opportunity for any organisation prepared to take it on?

In my view, unless and until something
like this is done, attempts by potential users to deploy RCF are likely to be constrained
by the limited availability of reference class data.

The basis upon which data could be
shared from a commercial standpoint is beyond the scope of this article. That
said, any concerns over confidentiality should be capable of resolution
through, for example, the anonymization of data and the appropriate
confidentiality agreements.

Overcoming resistance from
strategic manipulators

As already noted, some planning mistakes
will be genuine. Planners may not realise they are experiencing optimism bias. Where
this occurs the chances of getting buy-in to using RCF should be high. Why
would an honest planner want to achieve anything other than a better and
improved forecast?

On the other hand, where deception
or strategic manipulation occurs, there may be resistance to using RCF on a
project. If a planner’s inaccuracy is deliberate (to win funding or a new piece
of business) they may not welcome a process such as RCF if it is likely to reveal
the truth (ie that the working estimates are wrong and the project may fail as
a result). 

Wider adoption of RCF, and making it mandatory where appropriate,
should help overcome this challenge. Strategic
manipulators would then have no choice but to participate and abide by the
outcomes. 

Even
without blanket adoption there is nothing to prevent senior executives and
other decision-makers from insisting that RCF be used on a case-by-case basis. 

Risk of abuse?

There may be a risk that people
become complacent in situations where a project budget is increased or time-scales
extended as a result of RCF.  If there is
plenty of added contingency (to put it simply), project participants may simply
play to those upper limits rather than maintaining efficiencies and keeping
things tight. Is there a chance that the project will still overrun but against
higher thresholds or upper limits?

This is where the project managers
and lawyers (can and should) earn their keep. Good project management and the
inclusion of realistic and enforceable legal sanctions in contracts between the
parties to these projects should (in theory at least) help mitigate this sort
of situation.

Place more risk on contractors?

In a separate article published in
2014, Flyvbjerg and his co-authors[10] suggested
that ‘to ensure responsibility’ the
financial risk of delay and cost overruns should be placed with contractors who
bid on the project – in order to mitigate situations where the winning bidder submits
a low price in the expectation that they can make up the difference as the
scope increases.[11] They appear to suggest
that this risk allocation will incentivise contractors to be more honest at the
outset.

I agree with this – although the
probable reality is that a contractor (or supplier) will fight back and say
that the reason for the scope increase was changing requirements on the part of
the customer and not under-pricing when the deal was done. This is so often the
battleground in ICT legal disputes.

However, if it can be shown that
the original pricing and/or timescales (which tend to inter-relate) were always
wrong (based perhaps on RCF undertaken ‘after
the event’ to see what the original estimate should have been) then the contractor may be liable, particularly
if the original estimates were hopelessly understated and there is limited or
no evidence that a robust estimating process was conducted by the contractor when
preparing its bid.  In that instance, the
contractor could face a claim based on fraudulent misrepresentation.

But what happens in a
misrepresentation claim where the customer has undertaken RCF? Could this let
the contractor off the hook? Could it be said that the customer relied on its
own forensic estimates and not those of the contractor?

Is the position further
complicated if both parties performed RCF (independently or on a collaborative
basis) and agreed pricing and time-scales based on that information?

In these situations does it become
harder for customer organisations to argue for compensation (and any other
legal remedies sought) based on misrepresentation?  If so, should customers de-risk the situation
by leaving it entirely up to bidders to undertake RCF?  Or would that risk undermining the original purpose
of the exercise, namely for the customer, as the project initiator and
gate-keeper, to form its own views on time and cost to complete?

Conclusions

RCF is not (nor is it intended to
be) a ‘silver bullet’ solution to the problem of ICT project failure and there
are practical challenges associated with it. 
However the theory does make a lot of sense and one can see how it might
help improve outcomes, particularly when coupled with other (well-documented) good
practices such as:

(i) breaking up larger projects
into smaller, staged projects;

(ii) treating the project as a
business endeavour (not a technical one);

(iii) recruiting and keeping good
people to perform the work;

(iv) making people and
organisations accountable (including those engaged to perform RCF if they get
it wrong); and

(v) incentivising project
participants to be realistic and honest.

In summary, potential benefits
include:

– preventing parties from
embarking on unachievable projects – or at least giving them a better sense of
the actual risks involved so that adjustments can be made at the outset;

– lending more science to the art
of estimating;

– creating more certainty on scope
(ie the ability to set realistic requirements based on past similar projects);

– a transparent approach that
levels the playing field by, for example, making it harder for bidders to
under-price to win business; and

– fewer project failures.

Finally, in this age of big data,
surely there must be scope to collect, collate and organise historical ICT
project data from around the world and make it accessible to project planners
across the industry? In other words, if the ad hoc collections of project data
that currently exist can be pooled, shared and added to over time, RCF will
become more accessible and the benefits for the industry as a whole could be
spectacular.

Michael Bywell is a Partner at Arnold & Porter Kaye Scholer: Michael.Bywell@apks.com