Overconfidence and Calibration

The idea in plain words

A weather forecaster who says “seventy per cent chance of rain” is well calibrated if it rains on about seven of every ten such days. Most people are not like this about their own beliefs. Ask them for answers they are “ninety per cent sure” of, and they are right far less often than ninety per cent of the time. The gap between how sure we feel and how often we are right is overconfidence, and closing that gap is called calibration.

Calibration is a modest-sounding skill with large consequences. A team that is overconfident about its launch date, its market size and its competitive position will under-prepare for every way things can go wrong.

Where it comes from

The philosophical root is Socrates. In Plato’s Apology, Socrates explains why the oracle at Delphi called him the wisest of men. He had questioned politicians, poets and craftsmen, each of whom believed they knew important things, and found that their confidence outran their knowledge. He concluded that his only advantage was that he did not think he knew what he did not know. Socratic wisdom is calibration in its first form: knowing the limits of one’s knowledge.

The measurement came in the twentieth century. Weather forecasters were among the first to be scored. In 1950 the meteorologist Glenn Brier proposed a simple scoring rule for probability forecasts, now called the Brier score, which rewards both accuracy and honest probabilities. Psychologists then applied the same tests to ordinary people. Sarah Lichtenstein, Baruch Fischhoff and Lawrence Phillips reviewed the evidence in 1982: across many kinds of question, people’s stated confidence routinely exceeded their accuracy, most strongly on hard questions.

Philip Tetlock took the method to experts. From the late 1980s he collected tens of thousands of predictions from political and economic experts and scored them over many years. His 2005 book reported that, on average, the experts did little better than simple rules, and that the more famous and confident the expert, the worse the record tended to be. Borrowing a line from the poet Archilochus, made famous by Isaiah Berlin, he contrasted “hedgehogs”, who know one big thing and fit everything to it, with “foxes”, who know many little things and adjust. The foxes were better calibrated. Tetlock’s later forecasting tournaments showed that some ordinary people, working in teams and keeping score, could forecast far better than average, and that the skill could be trained.

What it does to the mind

Overconfidence has several sources. We remember our hits more than our misses. We construct coherent stories that feel more certain than the evidence supports. We rarely get clear feedback, because most predictions are vague enough to be reinterpreted afterwards. And within organisations, confidence is rewarded: the person who says “I’m sure” gets the budget, while the person who says “sixty per cent” sounds weak.

The result is a systematic gap between planning and reality. Launch dates slip, markets turn out smaller, competitors respond faster. Each is a separate surprise, but they share a cause.

Where it shows up

Several roads not taken, including Iridium and Itanium, rested on forecasts that were stated with great confidence and missed by an order of magnitude. Founders’ documents such as the Tesla Master Plan show the other side: public commitments made with confidence that the evidence of the time did not fully support, and which nonetheless helped make the outcome happen. The Decide cases ask readers to state a probability with each choice and keep a running calibration record.

What it asks of someone deciding

Use numbers. “Likely” means anything from fifty-five to ninety per cent depending on who says it. A probability forces precision and makes disagreement visible.

Keep score. Write down forecasts with dates and probabilities, and review them. Calibration cannot improve without feedback.

Distinguish the confidence you show from the confidence you hold. A leader may need to speak with conviction to a team. The planning documents behind that conviction should still carry honest probabilities.