When Small Risks Become Normal

The idea in plain words

The first time a seal on a rocket booster shows erosion after a flight, engineers are alarmed. They study it, conclude it was within acceptable limits and fly again. The next time, the erosion is a little worse, but the earlier analysis says it is acceptable, and so on. Each decision is reasonable given the one before. Years later, launching with eroded seals is simply how things are done, until the day the seal fails.

That is the normalisation of deviance: the gradual process by which an organisation comes to treat departures from safe standards as normal, because nothing bad has happened yet.

Where it comes from

The sociologist Diane Vaughan coined the phrase in her study of the 1986 Challenger disaster, published in 1996. The common story held that managers overrode engineers under schedule pressure. Vaughan found something more troubling: over years, NASA and its contractor had repeatedly observed damage to the booster joints and, through formal and careful processes, redefined it as an acceptable risk. The culture of production, the structure of information and the very rigour of the reviews made the fatal decision look normal to the people making it.

Her work built on two earlier traditions. Charles Perrow’s Normal Accidents (1984) argued that in complex, tightly coupled systems some accidents are inevitable. James Reason’s Human Error (1990) distinguished active errors at the front line from latent conditions built into systems by decisions made long before, the model often drawn as slices of Swiss cheese whose holes occasionally line up.

What it does to the mind

Success is the enemy of vigilance. Each time a risk is taken without disaster, it feels safer, though the probability may not have changed. Explanations that justified past decisions become premises for new ones, and dissent becomes harder because it implies that many previous decisions were wrong. People inside the organisation genuinely do not see the deviance, because they have watched it become normal one step at a time.

Where it shows up

The essay on enterprise software describes an institution that, over years, came to treat reports of unexplained shortfalls as evidence against the people reporting them, and large projects that continued long after warning signs. The essay on spreadsheets describes a risk model run on manual spreadsheets after reviewers had recommended automating it.

What it asks of someone deciding

Keep a written record of accepted anomalies and revisit it periodically with fresh eyes. The list itself makes drift visible.

Bring in outsiders who have not watched the risk become normal, and listen when they are alarmed.

Treat the absence of disaster as weak evidence of safety. A risk that has not yet caused harm may simply not yet have had its chance.