The Web That Would Understand Itself

The world it was born into

By the late 1990s the web had become the largest collection of documents ever assembled, and it was almost entirely written for people. A computer could fetch a page and display it, but it could not tell what the page meant: whether a number was a price or a date, whether a name referred to a person or a company, whether two pages described the same thing. Search engines were improving, but they matched words, not meaning.

Tim Berners-Lee, who had invented the web, saw this as its next frontier. In Weaving the Web (1999) he described a second phase in which data on the web would be described in ways machines could understand, so that software could help people do far more.

The bet

In May 2001 Berners-Lee, James Hendler and Ora Lassila published “The Semantic Web” in Scientific American. Its opening scenario imagined software agents arranging medical appointments for a family member by finding providers, checking insurance, comparing schedules and reasoning about travel time, all by reading machine-understandable information published across the web.

The technical plan had layers. Data would be expressed in a standard model, RDF, describing things and relationships as simple statements. Shared vocabularies, called ontologies and written in languages such as OWL, would define what terms meant and how they related. Logic and rules would let software draw conclusions. The W3C developed these standards through the 2000s, with substantial research funding in Europe and the United States.

Why it made sense

The problem was real. Combining information from different sources was, and remains, hard and valuable. The tools were grounded in decades of research in logic and knowledge representation. And the proposer had invented the web, which gave the vision unique credibility. In fields where data quality mattered and communities could agree on vocabularies, such as biology, medicine, libraries and government data, the standards proved genuinely useful.

How it lost

The general web did not follow. The vision required millions of publishers to describe their content precisely, using vocabularies agreed with others, and to keep those descriptions accurate. For most publishers the effort was real and the reward was distant, flowing mostly to the agents and users that would consume the data. Critics pointed out other difficulties early. Cory Doctorow’s 2001 essay “Metacrap” argued that people lie, are lazy and disagree about categories, so metadata would be unreliable. Clay Shirky argued in 2003 that the logical machinery assumed a tidy world of consistent facts that real information rarely provides.

Meanwhile, search engines solved many of the practical problems without asking anyone to change. They inferred meaning statistically, from links, words and user behaviour, across the web as it actually was. Their results were imperfect, but good enough, and they improved every year.

The vision survived where incentives aligned. Berners-Lee’s 2006 Linked Data note proposed simpler principles. DBpedia and Freebase built large structured datasets. In 2011 the major search engines launched schema.org, a simple shared vocabulary; publishers who added it to their pages got richer search listings. In 2012 Google launched its Knowledge Graph, built partly on Freebase, which it had acquired, and Wikidata launched as a structured companion to Wikipedia. A decade later, large language models began answering many of the questions the Semantic Web’s agents were meant to handle, without explicit ontologies or logic.

Explaining both the same way

It is common to say the Semantic Web was too complex and statistical search was simple. A symmetrical account notes that statistical methods were not simple at all; they required enormous computing resources and data, available only to a few companies. They won on the general web because that is where data was abundant and precision mattered less. The Semantic Web’s approach won where data was scarcer, precision mattered more and communities could agree, which is why it thrives in life sciences and libraries.

So the Semantic Web did not fail because explicit meaning is wrong. It aimed its approach at the problem where its costs were highest and its advantages smallest: the open, messy, commercially driven web, where nobody was paid to be precise.

What the loss teaches

The central lesson is about who pays. A system that depends on many people doing extra work needs to reward them directly and soon. Schema.org succeeded where the broader vision struggled because it gave publishers something they wanted in return.

The second lesson is about legibility. Formal categories make knowledge usable by machines, at the cost of forcing a messy world into shapes it does not naturally have. Statistical methods, and later machine learning, accepted the mess and learned from it.

Sources and limitations

This account relies on the 2001 Scientific American article, Berners-Lee’s writings and design notes, W3C standards dates and well-known contemporary critiques. Assessments of the Semantic Web’s success depend heavily on the domain: in some fields it is a quiet success, and proponents reasonably object to calling it a road not taken at all. The classification “solved the wrong problem” is the author’s interpretation, applied to its general web ambitions rather than to specialised uses.