How should we roll out a change that cannot fail?

The question behind the question

Executives usually treat a large implementation, a new enterprise system, a migration, a replacement of critical tools, as an operational matter for project managers. The record suggests it is one of the most consequential strategic decisions a company makes, because the failures are so severe. Of the decisions about executing large change on this site, nearly all backfired, and most were judged poor in process even by the standards of their time.

How they failed

FoxMeyer Drug, one of the largest drug distributors in the United States, installed SAP’s R/3 enterprise system and a new automated warehouse at the same time in the mid-1990s, and bid for contracts on the basis of savings the systems were expected to deliver. Neither worked as planned, and the company went bankrupt in 1996. Hershey Foods switched on new enterprise, customer relationship and supply chain systems together in July 1999, shortly before the Halloween ordering season; orders went unfilled during its busiest period. The US Air Force spent years and about a billion dollars on a single commercial system meant to replace hundreds of logistics systems while also changing its logistics processes, and cancelled it in 2012 without a usable result. Lidl adopted a standard retail system while keeping its own practice of valuing inventory at purchase prices, which required extensive customisation, and abandoned the project after seven years.

The worst case is not about cost. Britain’s Post Office treated data from its Horizon accounting system as reliable evidence against the sub-postmasters who ran its branches, prosecuting and pursuing hundreds of them for shortfalls between 1999 and 2015 that were in many cases caused by the system’s faults. The Court of Appeal later quashed many convictions, and a public inquiry has examined how the evidence of faults was missed. Two smaller cases make the same point about tools. JPMorgan’s Chief Investment Office ran a new risk model for a large derivatives portfolio on manually maintained spreadsheets in 2012, and an error contributed to losses of more than $6 billion. Public Health England loaded laboratory results into a spreadsheet format with a fixed row limit in 2020, and nearly 16,000 positive Covid-19 cases went unreported for days.

What the failures share

Four features recur. Several large changes were made at once, so that when something went wrong nobody could tell what. The go-live date was fixed to a business calendar or a contract rather than to readiness, and often fell at the worst possible time. There was no way back once the switch was made. And the organisation trusted the new system’s output over contrary evidence from people closest to the work, the problem the site’s idea Trusting the Machine Too Much describes, while small anomalies were explained away until they became normal, as in When Small Risks Become Normal.

The one success in the group is instructive in a different way. In 2020 the committee that names human genes renamed 27 of them, such as SEPT1 and MARCH1, because spreadsheet software kept converting their symbols into dates. Instead of fighting a tool used by millions of researchers, it changed what it controlled.

A diagnosis to start from

The diagnosis begins with how many things are changing at once, how reversible each change is, and when the business can least afford a failure. The guiding policy that follows is to separate changes, phase them, choose a go-live date for when failure can be absorbed, and treat the new system as unproven until its results have been reconciled with reality. The coherent actions include running old and new systems in parallel, piloting with a small part of the business, setting criteria in advance for rolling back, and creating a channel through which front-line staff can report that the system is wrong and be believed.

The strongest counter-case

The strongest objection is that phased change is slower and more expensive, that running parallel systems doubles work, and that some changes must happen at once because the parts depend on each other. A company that refuses to make large changes falls behind. All true. But the record does not argue against large change; it argues against making it in a way that removes the chance to notice and correct mistakes before they become catastrophes.

What the wider research says

Bent Flyvbjerg and Alexander Budzier studied 1,471 information technology projects and reported in 2011 that the average cost overrun was 27 percent, but that about one in six was a “black swan” with a cost overrun of around 200 percent and a schedule overrun of almost 70 percent. In How Big Things Get Done (2023), Flyvbjerg and Dan Gardner argued that successful large projects plan slowly and act quickly, use modular designs that can be built and tested in pieces, and take an outside view based on similar projects. Charles Perrow’s Normal Accidents (1984) explains why tightly coupled systems with complex interactions produce failures that no single error causes, which is what a big-bang implementation creates.