A Benchmark for Benchmark Clauses

The use of benchmark clauses in outsourcing contracts has grown rapidly over the last ten years.   Recently, however, benchmarking has been acquiring a bad name.  TPI, the outsourcing advisory firm, says: ‘Feedback received from both outsourcing clients and service providers is that the process of Benchmarking as a contractual means for testing and adjusting committed pricing is not working’.[1]  The problem is sufficiently bad that a recent sales presentation from a benchmarker opens with the statement ‘There is no doubt that benchmarking can be a long and trying process for both client and vendor’.[2]

I have had the unusual experience of working for three major benchmarkers (Gartner, Compass, and META Group) and also a large outsourcer (HP) during the last three years.  I have therefore seen benchmarking from several sides. 

I have come to realise that many of the difficulties in executing benchmarks are caused by the way benchmark clauses are written.  Very few of the lawyers or advisors who draft the clauses have been involved in actually executing a benchmark, and benchmarkers are almost never consulted about a clause before it is signed.  As a result, many contracts have been signed that benchmarkers find difficult to execute.

The purpose of this article is to feed back 10 years of benchmarking experience to lawyers and advisors.  I describe six essential factors to consider, and recommend the underlying principles behind each.  Note that these are my personal views, not the corporate position of Hewlett-Packard or any of my previous employers.  I believe them to be even-handed, fair, and constructive recommendations.  I wish to thank the many friends and colleagues who have reviewed my drafts.

Background

In a benchmark, an independent third-party specialist is engaged periodically during a contract.  They collect a comprehensive description of the services being delivered.  They consult a database of prices gathered from other contracts, and select a ‘peer group’ who are as similar as possible to the customer’s environment.  This group is used to calculate a target price which the outsourcer is expected to meet.

Benchmarking is re-assuring for customers.  It gives them confidence that they will continue to pay a fair market price throughout the duration of the contract, even if market prices change unexpectedly.  This re-assurance is essential if customers are to have the confidence to sign long-term agreements.

Benchmarking is not re-assuring for outsourcers.  From their perspective, a benchmark clause makes future profitability dependent on an unknown external organisation for whom there are no independent standards of good practice, and no proof of the accuracy of their methodology.  As a customer, you might be surprised that your outsourcer is particular about benchmarking.  As a share-holder in an outsourcer, it is the minimum you would expect.

Six Essentials

The worst benchmark clauses are simply incomplete, because they don’t clearly define what the benchmarker should do, or what happens after the benchmark.  I propose that all clauses should cover six key issues.

1. WHEN will the benchmark be done?

Underlying Principle: Fair Market Price

Benchmarking methods are designed to achieve a fair market price.  This means that the price is as good as could be achieved if the customer took their requirements to market tomorrow.  It also means that the outsourcer should make a reasonable profit, assuming that they have completed the hand-over and are working efficiently within any limitations set by the customer. 

Implications:

The clause should state when and how often benchmarking may be done.  It is almost impossible to benchmark an environment immediately after the contract start.  Benchmarkers work by taking a snap-shot of the services during a representative period, usually a month.  The information they need, such as the number of computers managed or the ownership of assets, changes greatly during transition and transformation.  Finding an accurate representative month is usually impossible.

Some clauses give the benchmarker a 30-day deadline to complete the project.  Benchmarkers find it extremely hard to achieve this.  Benchmarks typically take three months to execute, plus another one or two months to set up, and at least another month to implement the results.  Annual benchmarking is therefore semi-permanent benchmarking, and will constantly consume time from key people on both sides of the contract.  For most deals, benchmarking every two or three years is more practical. 

Contracts with committed year-on-year price reductions are to some extent pre-benchmarked, because the rate of price decline was presumably one of the criteria on which the outsourcer was selected over their competitors.  Benchmarking should be less frequent in these cases.  On contracts of three years or less with guaranteed year-on-year reductions, benchmarking should be unnecessary if the customer’s advisors have done their job well.

The costs of using a benchmarker are quite significant,[3] and can be hard to justify on contracts worth less than $1m per year.

 

2. WHO will do the benchmark?

Underlying Principles: Independence, Neutrality, Objectivity

Benchmarks should be carried out by independent third parties who have equal responsibilities to the two parties, and who are objective in their calculations.

Implications:

No-one knows who will have most benchmark data in the future years when a clause will be executed.  The best approach is therefore to choose the benchmarker at the time of the benchmark. 

Outsourcers expect that the benchmarker will be an independent consulting organisation, with a track record of such projects and a pre-existing database to draw from.  They also expect that they will not be benchmarked by their competitors.

It is reasonable for customers to ask for a list of suitable benchmarkers to be agreed in advance, to ensure that outsourcers cannot delay the process by challenging the selected benchmarker.

For IT outsourcing, there are two main benchmarkers.  Gartner Consulting is the largest, Compass is the second largest.  META Group are still proposed by some law firms, though they ceased to exist in 2005 and were absorbed into the Gartner Group.  New players are emerging, such as Maturity, Metri, Nautilus, and Global Information Partners. 

It is common to hire the benchmarker in a tri-partite contract in which the customer and outsourcer pay half of the benchmarker’s fees each.  Customers who expect to select and pay the benchmarker on their own should not be surprised if their outsourcer is suspicious about the benchmarker’s independence. 

 

3. HOW will it be done?

Underlying Principle: Accuracy

A benchmark must be accurate.  The benchmark clause should require the benchmarker to quantify the accuracy of their methodology, and should not be binding if this accuracy is not achieved.

Implications:

There are two main methods used by the benchmarkers.  When contractual benchmarking first started, nobody had databases of outsourced prices.  Benchmarkers therefore used a technique which I have called ‘cost-up’ benchmarking,[4] in which they calculate first the efficient delivery cost of the outsourcer, and then add an uplift to represent the overheads, risk, and margins which would be included in a market price.

Since 2000, some benchmarkers have built sufficiently large databases of contract pricing that they can now carry out direct price-to-price comparisons, subject to adjustments or normalisations to account for any differences between deals.  I call this ‘price-down’ benchmarking.

The merits of the two approaches are argued passionately within the benchmark community.  In practice, the best method is the one based most strongly on the underlying data, with the fewest estimates and adjustments.  This will vary from service to service, and will also change over time.  The benchmark clause should permit either approach.

While both customer and outsourcer need the benchmark to be accurate, most people accept that there are limits to accuracy.  Yet there are no agreed standards for acceptable limits, (For example, if one peer price is $30 per desktop and another is $200 per desktop, are the two services really comparable?  If the benchmark adjusts one of the peer prices by say 80%, is that too large to accept?)  Some clauses deal with this using a ‘tolerance’ statement (prices only change if they vary from the target by a pre-defined percentage), and this practice was recently recommended by TPI.[5]  Until such time as the benchmarkers make a public statement about the accuracy they achieve, the size of the tolerance must be individually negotiated. 

4. WHO will the prices be compared against?

Underlying Principles: Transparency & Confidentiality

It is reasonable for both customer and outsourcer to expect the benchmarker to show them how the conclusions have been reached, in as much detail as possible short of breaching the confidentiality of the peer group companies.  

Implications:

Benchmarkers promise both transparency and confidentiality.  Yet since these two fundamentally conflict with each other, in practice there must be limits to both.

Much of the information a benchmarker should ask for is easily available from the outsourcer, who should be obliged to provide it.  However, a benchmark should not require the outsourcer to open their books in a closed-book contract.  A benchmarker should be able to set a fair market price using only their database and the information provided to a customer by their outsourcer during the normal running of the operation.  (Open book or cost-plus contracts are an exception).  Should a customer wish such an internal inspection, an audit clause is an appropriate mechanism, not benchmarking.

The biggest concern amongst both customers and outsourcers is to ensure that the companies in the peer group are as similar as possible to the services being benchmarked.    It is common for benchmark clauses to state that the peer group members shall be from the same geography, same industry, same size, and same complexity.  These expectations, while understandable, make the benchmarker’s job extremely difficult.  The pool of data they can draw from is fixed – there is rarely enough time (or money) in a benchmark project to research new prices.  Selection of peer groups is always, therefore, a compromise.  Benchmarkers typically offer between six and ten companies in a peer group, with a minimum of four, and rarely more than twelve.  The more companies that are included, the more differences there will be between some of them and the customer’s environment.

I recommend that the benchmark clause should require peer group members to receive services with similar Scope, Scale, Service levels, Sites, and Special contractual conditions.  The ‘Five S’ test will enable the benchmarker to select the best group they can.

Outsourcing contracts are normally performed under conditions of strict confidentiality.  Customers want to ensure that the prices they have negotiated do not reach their competitors.  Outsourcers feel the same way.   Yet a benchmarker needs to use the information they collect on one project on the next project.  This paradox can be resolved in three ways: by keeping the names of the peer group members secret, by limiting the detail about the peers which is disclosed, or by presenting the prices of individual peers without showing which peer pays which price. 

Outsourcers usually like to see as much detail as possible, to ensure that no mistakes were made in the calculations.  Benchmarkers usually resist this, citing either the anonymity of the peer group companies or a desire to protect their intellectual property and proprietary methodologies.  Unspoken, the benchmarkers sometimes fear that the outsourcer is probing for weaknesses which can be used to undermine the benchmark.  They ask the customer and outsourcer to ‘trust them’.  Outsourcers feel this is evasive -– they expect that trust must be earned.

The outsourcer’s acceptance of the report will be increased if the benchmark clause provides for a review of the draft results.  A period of at least two weeks should be reserved for both parties to make comments for the benchmarker to consider before the final report is delivered.   Most benchmarkers encourage this practice.

 

5. HOW will the target price be calculated?

Underlying Principle: Groups of Peers

Benchmarking techniques set fair market price based on a group of companies, not based on a single organisation.  

Implications:

Given a free choice, all the major benchmarkers now use the average (arithmetic mean) of the adjusted peer group prices to set the target price.    

The problem is, every customer wants to be better than average.  In the past, many clauses have used a percentile target, usually the 25^th percentile, as a way of expressing this.[6]  This technique is now widely discredited.[7]  Percentiles are a statistical technique intended for use with large numbers of data points.  Applied to a small group of less than ten peers, the 25^th percentile in practice is set almost entirely by the second highest peer’s prices.  Attempts to apply the 25^th percentile to service levels make matters worse – for example: it is mathematically wrong to add 99.9% availability to a four hour problem resolution time and then calculate the 25^th percentile.

Purchasing managers sometimes expect that a benchmark can ensure that a customer receives the lowest price being paid for a service anywhere in the world.  It can’t.  It is unlikely that any contractual provision can guarantee this – even Most Favoured Customer clauses are controversial, difficult to execute and almost impossible to police.[8]

6. WHAT will happen after the benchmark?

Underlying Principles: Forward-Looking, Binding, and Constructive

Following the benchmark, both outsourcer and customer should be bound to a specific, timed plan to make changes which best improve the future operation of the contract.  Benchmarks cannot change the past, and should not try to reverse past pricing, investments, or commitments.

Implications:

Most customers see the primary goal of a benchmark as price reduction, incentivising outsourcers to improve efficiency and productivity.  Most outsourcers see this as one-sided, and would like a benchmark to either raise or reduce prices, quoting increases in the cost of labour, real estate and energy as justification.

The benchmark clause should not try to second guess the customer’s needs at the time of the benchmark.  Of course they may want lower prices, but it can also happen that they are more interested in improved service levels at the same price.  The clause should permit this as an option.

Customers also want price changes to be made without delay.  However, benchmarks usually express recommendations as adjustments to the total charges, leaving details to be negotiated such as changes to individual service prices, the knock-on impact on volume breaks (ARCs/RCCs) and the consequences for revenue commitments.  It is reasonable to spend a month agreeing the details and making any necessary engineering and contract changes.  It also simplifies life greatly for everyone who works on the contract in future if any price changes are made at the start of a billing month, not a fixed number of days after the arbitrary date when the benchmark report happens to be presented.

Some contracts seek to avoid sudden large price changes which might destabilise the services by including a ‘limiting’ statement (a pre-defined maximum percentage that prices can change in one year).  This practice is recommended by TPI, and re-assures outsourcers who are concerned about the accuracy of benchmarking. 

Retrospective back-dating of benchmark charges is best avoided.  Benchmarks are nearly always done as a snap-shot of the current year, not taking into account previous years.  The few clauses signed in the past that included back-dating generated acrimonious and painful benchmarks, which can end in early termination and even litigation.

Equally important is the question of what happens if the outsourcer cannot meet the benchmark targets.  It is reasonable for the customer to expect that the clause ‘has teeth’, and that, if the outsourcer cannot provide the benchmarked services at the benchmark price, they can be acquired from elsewhere.  In some deals, however, this would mean that the outsourcer would be faced not only by loss of revenue but also by the loss of investments made at the start of the contract, such as purchase of assets, employment guarantees, or amortised transition/transformation costs.  These investments are made based on commitments by the customer to revenue levels and/or contract lengths.  If the customer sees the benchmark clause as a way of avoiding these commitments, they raise the level of risk substantially for the outsourcer, and they should expect this to be reflected in the price they pay.

It is unreasonable to expect an outsourcer to be in material breach of a contract because they have failed to meet a target which has not yet been given to them, set by an organisation which has not yet been selected.

Good Practice In Benchmark Clauses

When I started to write this article, I hoped to be able to draft an industry standard benchmark clause.  I no longer believe this is possible, and have lowered my ambitions to the list of ‘essentials’ above.  I can also offer, however, three further suggestions which, while not essential, will help produce a workable benchmark clause.

A. Cross-References

It is unlikely that a benchmark clause can exist in isolation from the rest of the contract.  Ensure there are cross-references to other sections.  For example how will Cost of Living adjustments be applied to the revised benchmark price?  If year-on-year price reductions are promised, which takes priority: the reductions or the revised benchmark price?  In the unhappy event of an outsourcer not meeting the benchmark price and the customer withdrawing from the agreement, which termination clause shall apply?  (It is common that neither Cause nor Convenience are suitable, and a third option is needed.) Finally, if either party feels that a benchmarker has made a mistake, what process shall be followed?  Advisors fear that such a process can be used to undermine the benchmarking process.  Yet for both customer and outsourcer, benchmark recommendations will override the carefully constructed plans for cost and revenue on which the contract was based.  It is not unreasonable to ask for safe-guards.

B. Agreement To Agree

Central to the methods of all the benchmarking companies is a desire to ensure that all the parties agree at each step of the process.  It is no good arriving at a benchmark price and then finding that the data input to the benchmark was inaccurate, or that some of the peers are not acceptable.  A benchmark clause should express this principle.  I have seen legal reviews remove this language on the grounds that it is an agreement to agree.  A benchmark clause is always, in some ways, an agreement to agree.   Some organisations expect that a benchmark will be like consulting an inflation indicator from an official government publication.  In practice, it is much closer in nature to mediation.

C. Length

Benchmark clauses get longer every year.  This is not a good thing.  It is difficult and unwise for the benchmark clause to try to define every aspect of a process which will be carried out several years in the future.  As long as the essentials above are covered, other decisions should be left to the three parties at the time.  A complete benchmark clause can be written on less than two sides of paper.

Conclusions

For advisors and lawyers who draft benchmark clauses, this article offers a checklist for structuring them.  They can at last apply the experience of benchmarkers to the construction of a benchmark clause. 

Improvements in the industry will only happen slowly, however.   If you recently signed a contract with a bad benchmark clause then you may well find, as others have done, that your benchmark does not run smoothly.  You must either live with this clause for the life of the contract, or agree to re-negotiate it.  If you re-negotiate, this article sets out standards that your new clause should meet so that benchmarking improves your sourcing relationship to the benefit of both parties. 

Neil Barton specialises in measuring the performance of corporate IT organisations. He has worked at benchmarkers Compass, META Group, and Gartner, and is currently Manager of Benchmarking at Hewlett-Packard Services: www.neilbarton.com