Superforecasting

The model

Philip Tetlock spent nearly two decades, from the mid-1980s, collecting tens of thousands of predictions from experts in politics and economics and checking them against what happened. The results, published in 2005, became famous in caricature: the average expert did about as well as a dart-throwing chimpanzee. The fuller finding was more interesting. Some experts did better than others, and the difference depended less on what they knew than on how they thought. Borrowing an image from the philosopher Isaiah Berlin, Tetlock called the better forecasters foxes, who know many things, and the worse ones hedgehogs, who know one big thing and fit everything to it.

In 2011 the US intelligence community’s research agency, IARPA, launched a forecasting tournament to find out whether prediction could be improved. Tetlock and Barbara Mellers led one of the competing teams, the Good Judgment Project, recruiting thousands of volunteers to forecast questions such as whether a particular government would fall or whether a commodity price would exceed some level within a set period. Their team won decisively over several years. Superforecasting, written with the journalist Dan Gardner and published in 2015, describes what they learned, especially about the small group of volunteers, around the top two per cent each year, who forecast best: the superforecasters.

The model has three parts.

Measure. Forecasts must be precise enough to be scored. Vague statements such as “a serious possibility” mean different things to different people and can never be proven wrong. The Good Judgment Project asked for numerical probabilities on questions with clear deadlines and outcomes, and scored them with the Brier score, which measures both calibration, whether events given 70 per cent happen about 70 per cent of the time, and resolution, whether a forecaster can tell likely events from unlikely ones.

Think like a fox. Superforecasters share a style of thinking more than any particular knowledge. They break hard questions into smaller parts that can be estimated separately, a technique the authors call Fermi-izing after the physicist Enrico Fermi. They begin with the outside view, asking how often things like this usually happen, before adjusting with specific information. They gather perspectives from many sources and synthesise them, like the compound eye of a dragonfly. They update frequently, in small increments, as new information arrives, avoiding both under- and over-reaction. And they treat their beliefs as hypotheses to be tested rather than treasures to be guarded.

Work as a team and practise. Superforecasters improved further when they worked in teams that shared information and challenged each other, and the project found that a short training module in probabilistic reasoning measurably improved accuracy. Forecasting, on this view, is a skill like any other: it improves with practice and feedback.

Seeing with it

The model explains why so much commentary is useless as forecasting. Pundits make vague predictions, are rarely scored, and are rewarded for confidence and a good story rather than accuracy. Organisations make the same mistakes in strategy documents, where “likely” and “possible” are never defined and nobody checks back.

It also explains how to do better in practice. Before predicting whether a new product will reach a sales target, a forecaster in this style would ask how often comparable products reach comparable targets, then adjust for what is specific about this one, then put a number on it, then update as data comes in. The site’s idea pages The Inside View and the Outside View and Overconfidence and Calibration explore these habits.

The book gives a humane picture of expertise too. Superforecasters were retired engineers, pharmacists and filmmakers, not famous analysts. What they had in common was curiosity, numeracy, humility about their own knowledge and a willingness to work hard at being right.

Where the model stops

The authors are clear that the tournament questions had horizons of months to a year or two. Accuracy falls sharply further out, and the method says little about predicting events years ahead. Nassim Taleb has argued that the most consequential events are rare, extreme ones that tournaments of this kind cannot score well, and that what matters for decisions is exposure to such events rather than their probability. Tetlock engaged with this criticism, and the two co-wrote a paper on the difference between forecasting binary outcomes and managing fat-tailed risks.

The model also measures accuracy on questions someone has already posed. Choosing the right questions, those that matter for a decision, is a separate skill the tournament did not test.

Since it was published

The Good Judgment Project became a commercial company, Good Judgment Inc., which offers forecasts and training. Forecasting platforms such as Metaculus and prediction markets have spread the practice of public, scored probabilistic forecasting. Research on forecasting has continued, including studies of how to combine individual forecasts, and recent work has compared human superforecasters with large language models on the same kinds of questions. The basic lesson, that keeping score is the start of improvement, has spread into intelligence, policy and business planning.

Questions it lets you ask

What exactly are we predicting, by when, and how will we know whether we were right? What is the base rate for events like this? What number would we put on it, and what would make us change that number? Who disagrees with us, and what do they see? Are we updating as information arrives, or defending our first answer? And are we keeping a record of our forecasts so we can learn from them?