Trusting the Machine Too Much

The idea in plain words

A pilot follows the autopilot’s altitude reading even as the view outside suggests otherwise. A clerk accepts a system’s figure that a customer owes money, though the customer insists they paid. A writer accepts an AI’s confident citation without checking it. In each case the person defers to the machine.

Automation bias is the tendency to favour suggestions from automated systems and to discount contradictory information. It has two forms: errors of commission, following a wrong recommendation, and errors of omission, failing to notice a problem because the system did not flag it.

Where it comes from

In 1983 the psychologist Lisanne Bainbridge published “Ironies of Automation,” noting that automating a process leaves people with the hardest tasks, taking over when the automation fails, while depriving them of the practice they need to do it. In the 1990s Kathleen Mosier and Linda Skitka studied pilots and students using automated aids and documented the errors of commission and omission. Raja Parasuraman and Victor Riley’s 1997 paper classified how people use, misuse, disuse and abuse automation, and became a foundation of the field.

What it does to the mind

Deferring to automation saves effort, and people are economical with effort. A system that is usually right teaches its users to stop checking, so their vigilance decays exactly as their dependence grows. Machine outputs also look objective: a number on a screen carries no trace of the assumptions or errors behind it. And responsibility diffuses, since a person who followed the system feels less accountable than one who chose alone.

Where it shows up

The essay on enterprise software describes the Post Office prosecuting hundreds of sub-postmasters on the evidence of an accounting system that a court later found contained bugs, errors and defects, in a legal setting that presumed computers worked correctly. The essay on large language models describes fluent, confident answers that are sometimes wrong, and the essay on spreadsheets describes models whose errors went unchecked because their numbers looked authoritative.

What it asks of someone deciding

Treat system output as a claim, not a verdict. Build ways to examine how a figure was produced, and processes for investigating when people dispute it.

Look for patterns in overrides and complaints. One person disputing the system may be mistaken; many people reporting the same anomaly is evidence.

Preserve the skills needed to catch errors, through practice, rotation and occasional manual work, especially as AI takes on more tasks.