Silent Evidence

The idea in plain words

If you study only the companies that became giants, you will find that they took big risks, ignored the sceptics and stayed committed to a vision. You may conclude that taking big risks, ignoring sceptics and staying committed are how companies become giants. But the graveyard is full of companies that did exactly the same things. They are not in the study because they are no longer around to be studied.

This is survivorship bias, or what Nassim Nicholas Taleb called silent evidence: the evidence that would overturn our conclusions is precisely the evidence that has been removed from view.

Where it comes from

The oldest version of the argument is a story told by Cicero about Diagoras, a Greek thinker known for doubting the gods. A friend shows him the votive tablets in a temple, painted by sailors who prayed during a storm and survived. Surely this proves that prayer saves people at sea? Diagoras replies that there are no paintings of those who prayed and drowned. In 1620 Francis Bacon retold the story in the Novum Organum as an example of how the human mind seizes on agreeing cases and neglects the ones that would contradict it.

The modern version comes from the Second World War. Abraham Wald, a mathematician at the Statistical Research Group at Columbia University, worked on how to estimate the vulnerability of aircraft from the damage seen on planes that returned. His memoranda of 1943 developed the statistical logic: the returning planes were a biased sample, because planes hit in the most vulnerable places did not return. Damage that appears rarely on survivors may indicate exactly where hits are fatal.

The popular retelling, in which Wald looks at a diagram of bullet holes and tells the military to armour the parts without holes, is a later simplification. His actual work, reconstructed by Marc Mangel and Francisco Samaniego in 1984, was a set of technical methods for estimating hit probabilities, not a single anecdote. The simplified story is itself a small lesson in how a tidy narrative replaces the messier record.

What it does to the mind

Survivorship bias works through availability. The mind judges what is common or effective by what comes easily to mind, and survivors are what comes to mind. Successful founders give interviews; failed ones rarely do. Products that won are documented; products that lost leave a few press releases and some forum posts. The organisational theorist Jerker Denrell showed in 2003 how this undersampling of failure produces management myths: practices that look effective because the firms that tried them and failed have dropped out of the sample.

There is also a motivational element. Survivors’ stories are flattering. They suggest that success is earned by qualities anyone could choose to have, which is a more comforting picture than one in which luck and timing play a large part.

Where it shows up

The whole Roads Not Taken section is an attempt to put the silent evidence back. Betamax, Xanadu, Lisp machines and Iridium were serious projects backed by capable people, and several of them looked more likely to win than their eventual rivals. Without them, the histories of video, hypertext, computing and mobile phones would look like straight lines towards the obvious.

Note-taking apps and email clients, both discussed in essays here, have unusually large graveyards. The surviving products look like natural winners, but the note-taking essay and the email essay both describe products that had the same ideas and did not last.

What it asks of someone deciding

When someone recommends a strategy because a famous company used it, ask for the denominator. How many companies tried it? A practice shared by every winner may also be shared by most losers.

Look for failure data deliberately: start-up post-mortems, discontinued products, abandoned projects inside your own company. They are harder to find and less pleasant to read, which is why they are valuable.

Be careful with your own history. The projects you remember are the ones that shipped. The lessons you draw from them will be skewed unless you also remember the ones that were quietly cancelled.