The Auction Behind the Answer

Every day several billion people type a few words into a box and receive, in under a second, an ordered list. The list is so familiar that it hardly seems like a decision. Yet each ranking embodies a judgement about what is relevant, whose page deserves attention, what a person probably meant and what they will probably do next. And alongside the list, labelled but adjacent, is another ranking: of advertisers who have bid, in an auction that took place in the same fraction of a second, for the right to be seen by this person at this moment.

Web search is the first thing many people do when they do not know something, and the main way most of the web is found. It is also the most profitable business model the media industry has ever produced. In 2025 Alphabet, Google’s parent company, reported revenue of $402.8 billion, and in the last quarter of that year alone “Google Search & other” brought in about $63 billion.¹ Almost all of that money comes from advertisers paying to be found.

This essay follows how search came to work this way. Its story runs from library catalogues to citation indexes, from hand-built directories to algorithms that read the structure of the web, from a disreputable idea for selling rankings to the auction that funded the free internet, and from ten blue links to a paragraph written by an AI. Its deeper subject is relevance: what it means, who decides it and what happens when the people who want to be found are the ones paying for the decision. The founders of the most successful search engine saw that problem clearly before they built a business on it. Their warning is the right place to begin.

“Inherently biased towards the advertisers”

In April 1998 Sergey Brin and Lawrence Page, two graduate students at Stanford, presented a paper at the World Wide Web conference in Brisbane describing a prototype search engine called Google. Most of the paper is technical: how to crawl the web, how to store an index of 24 million pages, how to calculate a ranking from the links between them. At the end, in an appendix titled “Advertising and Mixed Motives,” the authors stepped back. The predominant business model for commercial search engines was advertising, they wrote, and “the goals of the advertising business model do not always correspond to providing quality search to users.” They gave an example from their own prototype: for the query “cellular phone,” one of the top results was a study of the dangers of using a phone while driving, a page a search engine paid by phone advertisers would have difficulty justifying. “For this type of reason and historical experience with other media,” they concluded, “we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.”²

They went further. Because even experts find it hard to evaluate a search engine, bias in search is “particularly insidious.” They cited reports that a company called OpenText had sold placements at the top of its results, which had caused an uproar, and noted that subtler bias, such as adding a small factor to “friendly” companies, would be very difficult to detect. They observed that a major engine did not return a large airline’s home page for a search on the airline’s name, because the airline had bought an expensive ad linked to that query. A better engine, they pointed out, would not have needed the ad, and would have lost the revenue.

Within four years the company those two students founded had built the largest advertising business on earth on top of search. That is not necessarily hypocrisy. The interesting question is what they and their colleagues designed in order to make an advertising-funded search engine less biased than the one they feared, how well it worked and where it eventually failed. To answer it, we need to go back further, to the older problem that search was built to solve.

Before the web: the problem of finding

The problem is simple to state and hard to solve. There is a collection too large for anyone to read, and a person who needs something in it. How does the person find it?

Libraries had answered for centuries with catalogues. In 1876 the American librarian Charles Ammi Cutter published rules for a dictionary catalogue that set out its purposes in terms still taught: to let a reader find a book when they know its author, its title or its subject, to show what the library has by a given author or on a given subject, and to help the reader choose among editions.³ Catalogues relied on human judgement at the point of entry. A trained cataloguer read each book and decided what it was about, using a controlled vocabulary of subject headings. The reader then had to guess the heading the cataloguer had chosen.

In the first half of the twentieth century a few visionaries imagined something more ambitious. The Belgian bibliographer Paul Otlet built, from the 1890s onward, a vast card index of the world’s published knowledge in Brussels, and wrote of a future in which anyone could consult it from a distance. In 1945 the American engineer Vannevar Bush, who had directed the United States’ wartime research effort, published “As We May Think,” which imagined the memex, a desk on which a scholar could store records and connect them by “associative trails,” following the way the mind jumps from one idea to the next.⁴ The paper on As We May Think examines how much of it came true. Its importance here is that it described finding as following connections, not consulting an index.

After the war, as scientific literature multiplied, a new field of information retrieval began to treat finding as a problem for experiment and measurement. At the College of Aeronautics in Cranfield, England, the librarian Cyril Cleverdon ran experiments in the late 1950s and 1960s that compared indexing methods against fixed collections of documents, fixed questions and human judgements of which documents were relevant. The Cranfield method, which measures a system by its precision (how many retrieved documents are relevant) and recall (how many relevant documents are retrieved), remains the basis of how search engines are evaluated. At Harvard and then Cornell, Gerard Salton’s SMART system represented documents and queries as lists of weighted words and ranked documents by their similarity to the query. In 1972 Karen Spärck Jones, working in Cambridge, showed that a word that appears in few documents is much better evidence of relevance than a word that appears in many, and proposed weighting terms accordingly.⁵ Her idea, later called inverse document frequency, combined with counts of how often a word appears in a document, became the standard way of scoring text for decades. Spärck Jones also argued that the specificity of a term should be measured statistically, from how words are actually used, not from what they mean, an early sign of a long shift from meaning to statistics.

A different line of thought came from the study of science itself. In 1955 the information scientist Eugene Garfield proposed in Science a citation index: a directory showing, for each published paper, every later paper that cited it. A reader who had found one useful paper could then follow citations forward in time to the work that built on it.⁶ Garfield’s Science Citation Index appeared in the 1960s, and with it came ways of measuring a paper’s or a journal’s influence by how often it was cited. Later bibliometricians, notably Gabriel Pinski and Francis Narin in the 1970s, weighted citations by the influence of the citing journal, so that a citation from an important journal counted for more. The intuition that a document can be judged by who refers to it, and that references from important documents matter more, would reappear on the web almost unchanged.

By the 1980s, then, information science had three ideas that search engines would combine. Relevance could be estimated statistically from the words in a document. Systems could be evaluated by comparing their rankings with human judgements. And the structure of references between documents carried information about their importance that their words did not.

The first engines: lists, directories and full text

The internet before the web was a collection of servers offering files. In 1990 Alan Emtage, Bill Heelan and Peter Deutsch at McGill University in Montreal wrote Archie, a program that regularly collected the file listings of public FTP servers so that a user could search for a file by name.⁷ Archie did not read the files; it indexed their names. Similar tools followed for Gopher, a menu-based system popular in universities.

When the World Wide Web spread from 1993, the first serious attempt to organise it was human. In early 1994 Jerry Yang and David Filo, two Stanford graduate students, began keeping a list of their favourite websites, sorted into categories. The list became Yahoo!, incorporated in 1995, with editors who reviewed sites and placed them in a hierarchy of topics, much as a library classified books. A directory was intelligible and trustworthy: a person had looked at every site in it. It could not keep up. The web grew from a few thousand sites to millions within a few years, and no team of editors could read them all.

The alternative was to send a program, a crawler or spider, to follow links from page to page, download each page and build an index of every word on it. WebCrawler, created by Brian Pinkerton at the University of Washington, indexed the full text of pages in 1994. Lycos, from Carnegie Mellon, followed the same year. In December 1995 Digital Equipment Corporation launched AltaVista, built by researchers at its Palo Alto laboratory, including Louis Monier and Michael Burrows, partly as a demonstration of the power of its Alpha computers.⁸ AltaVista was fast and comprehensive, and for a few years it was the engine that serious users preferred. Excite, Infoseek, HotBot and others competed with it.

These engines ranked results mainly by the words on the page: how often the query words appeared, whether they appeared in the title, how close together they were. The method inherited from information retrieval worked well for collections of scientific abstracts written in good faith. It worked badly on the web, for a reason the information scientists had rarely had to consider. Web authors wanted to be found, and they could see how the ranking worked. Pages repeated popular words hundreds of times, sometimes in text the same colour as the background. Ranking by words rewarded whoever was most willing to manipulate them. By 1997 searches on popular terms on the leading engines often returned pages that matched the words and had nothing useful to say.

The leading companies also had a business problem. Search was expensive to run and did not obviously make money. The answer most of them chose was to become portals: home pages offering news, email, weather, shopping and chat as well as search, so that users would stay and see more banner ads, sold by the thousand impressions. In that model, search was a feature whose job was to keep users on the portal, and a search engine that sent users away quickly was failing at its commercial purpose.

Links as votes: PageRank

Several people saw at about the same time that the web’s links could solve the ranking problem. In 1996 Robin Li, working for a company in New Jersey, built a system called RankDex that scored sites by the links pointing to them, and filed for a patent the following year; he later founded Baidu, China’s leading search engine. In 1998 Jon Kleinberg, then at IBM, published the HITS algorithm, which distinguished “authorities,” pages that many good pages linked to, from “hubs,” pages that linked to many good authorities.⁹

At Stanford, Larry Page had begun a project called BackRub to map the links of the web, and with Sergey Brin turned it into a ranking method. Their 1998 paper explained the intuition plainly. “Academic citation literature has been applied to the web, largely by counting citations or backlinks to a given page,” they wrote; PageRank extended this “by not counting links from all pages equally, and by normalizing by the number of links on a page.”² A page was important if important pages linked to it. Because that definition is circular, the ranking was computed by repeated approximation across the whole map of links, which for their prototype contained up to 518 million hyperlinks.

They also described PageRank as a model of behaviour. Imagine “a ‘random surfer’ who is given a web page at random and keeps clicking on links, never hitting ‘back’ but eventually gets bored and starts on another random page.” The probability that this surfer is on a given page at any moment is its PageRank. The model made the ranking intuitive: pages that many paths through the web lead to are the pages people are likely to find.

The innovation was not only the mathematics. It was the choice of evidence. A word on a page is written by the page’s author, who may want to be found. A link from another page is written by someone else, for their own reasons, usually to point their readers to something they considered useful. Links were, in 1998, a largely honest signal: a byproduct of millions of people making judgements about what was worth reading. Google combined PageRank with traditional text scoring, with the text of the links themselves (anchor text, which often describes the target page better than the page describes itself) and with other signals. The results for common queries were visibly better than those of its rivals.

Brin and Page tried to license their technology to existing search companies and found no buyer at a price they would accept. Accounts differ on the details, but the established engines were then focused on becoming portals, and a better ranking algorithm did not seem worth much. In August 1998 Andy Bechtolsheim, a founder of Sun Microsystems, wrote them a cheque for $100,000, and Google was incorporated in September. Its home page was almost empty: a logo, a box and two buttons. That spare design was partly a matter of limited resources. It also expressed a different theory of the product. Google wanted users to leave its page as fast as possible, because that meant they had found what they wanted. From its early years the company described its mission as organising “the world’s information” and making it “universally accessible and useful,” a librarian’s ambition stated as a company’s purpose.

Selling the ranking: GoTo’s auction

At almost the same moment, a company in Pasadena took the opposite view. Bill Gross, who ran an incubator called Idealab, observed that the most valuable thing a search engine knew was what its users wanted, expressed in their own words at the moment they wanted it. Someone searching for “Hawaii vacation” was a better prospect than anyone who saw a banner ad on a news page. In February 1998 GoTo.com began letting advertisers bid for the positions in its results for particular search terms. The highest bidder appeared first, and the advertiser paid its bid each time a user clicked on its listing.¹⁰

Several features of GoTo’s design were new and lasting. Advertisers paid for results, clicks, not for exposure. Prices were set by an open, continuous auction, not by a sales team. And the auction worked for any word in the language, so that a florist in a small town could bid on “flowers” and that town’s name and pay a few cents, while an insurance company paid dollars for “car insurance.” Early bids were a cent per click; within months some terms commanded a dollar. GoTo turned search queries into a market in which commercial attention was priced word by word.

GoTo also took a position that the Stanford paper had warned against: its ranking was the auction. Advertisers who paid more appeared higher, and GoTo did not hide this; it argued that a business willing to pay for a click was likely to be relevant to the query. Critics, and Brin and Page, saw it differently. The company’s own search site never became popular. Its success came from supplying paid listings to the portals, to AOL, to Microsoft’s MSN and to Yahoo!, which placed them at the top of their own results. Renamed Overture in 2001, it was by then a highly profitable business and was granted a US patent on its method of bid-for-placement listings.¹¹

Google’s synthesis: AdWords

Google’s founders were at first wary of advertising, and the company tried selling search technology to companies and portals. In June 2000 it won the contract to provide web search for Yahoo!, then the most visited site on the internet, which displayed “Powered by Google” on its results. In October 2000 Google launched AdWords, a self-service system for small text ads that appeared beside, not within, its results. It began with about 350 advertisers, and at first charged them by the number of times their ads were shown.¹²

In February 2002 Google relaunched the system as a cost-per-click auction, adopting the model that Overture had shown advertisers would accept. But it changed the auction in three ways that mattered.

First, it did not rank ads by bid alone. An ad’s position depended on its bid multiplied by an estimate of how likely users were to click on it. An advertiser who bid $1 for an ad that users clicked twice as often as a rival’s could outrank a rival who bid $1.50. Google later broadened this into a “quality score” that also considered the relevance of the ad and of the page it led to. The economist Hal Varian, who joined Google in 2002 and became its chief economist, explained the logic: ranking by bid times click-through rate ranks ads by the revenue they are expected to generate, and ads that users find useless sink, which protects the value of the whole page.¹³

Second, an advertiser did not pay its bid. It paid the minimum amount needed to keep its position above the next advertiser, plus a cent. This “generalised second-price” design reduced the incentive to keep adjusting bids in small steps to undercut rivals, a pattern that had produced unstable, cycling prices in first-price auctions like Overture’s. Economists Benjamin Edelman, Michael Ostrovsky and Michael Schwarz later showed that this auction is not strictly “truthful,” meaning that bidding one’s true value is not always the best strategy, but that it has a stable outcome closely related to the theoretically ideal auction.¹⁴ The design was not derived from theory; it was arrived at by engineers, notably Eric Veach and Salar Kamangar, and economists then explained why it worked.

Third, and most important for trust, Google kept the ads separate from the organic results and labelled them. The organic ranking, determined by PageRank and other relevance signals, could not be bought. Ads appeared in a shaded box above the results and in a column to the right. That separation was the company’s practical answer to the problem its founders had described in 1998. Advertising would fund the engine, but it would occupy its own space, under its own label, with its own auction, and would not move the ranking beside it. The US Federal Trade Commission, responding to a complaint from a consumer group, wrote to search companies in June 2002 that paid placement should be clearly distinguished from natural results, which helped make Google’s practice the industry norm.¹⁵

Overture sued Google in April 2002 for infringing its patent. The dispute was settled in August 2004, just before Google’s public offering, together with a separate dispute over share warrants, when Google issued 2.7 million shares to Yahoo!, which by then owned Overture, in exchange for a perpetual licence.¹¹ Overture had invented the business model. It lost the market to a company that had combined the business model with a better search engine and a better auction.

In 2003 Google extended the auction beyond its own pages. It bought Applied Semantics, a company whose technology could identify what a web page was about, for about $102 million, and turned it into AdSense, a system that placed auctioned ads on other people’s websites and shared the revenue with them.¹⁶ For millions of small publishers this was the first practical way to earn money from writing on the web. It also put Google on both sides of a delicate relationship: its search ranking decided which pages received traffic, and its ad network paid many of those same pages for the traffic they received.

The portal that outsourced its future

Yahoo!’s decision to use Google’s results in 2000 deserves attention because it is one of the clearest strategic errors in the history of the web, and because it was not obviously an error at the time. Yahoo! was a media company. It thought of search as one of many services, and Google as a supplier. The arrangement gave Yahoo! good results cheaply.

It also gave Google a large share of the web’s queries, the data that came with them and a credit line on the most visited page in the world. As users noticed that the quality came from Google, many went straight to the source. Yahoo! responded by buying Inktomi, a search technology company, in 2003, and Overture the same year for $1.63 billion. In 2004 it replaced Google’s results with its own.¹⁷ By then Google was already the leading search engine and the leading search advertising business. Yahoo!’s own improved ad system, launched in 2007 under the name Panama, added the quality weighting that Google had used since 2002. In 2009 Yahoo! agreed to use Microsoft’s new Bing engine for its search.

The lesson is not that Yahoo!’s managers were foolish. It is that search turned out to be a different kind of business from the portal. A portal competed for time spent on its pages. Search competed for intent, the moment when a person declared what they wanted, and intent was what advertisers would pay most for.

The arms race for the first page

Once Google’s ranking decided where most of the web’s readers went, the ranking itself became a market. Search engine optimisation, the craft of making pages rank higher, grew into an industry. Much of it was legitimate: clear titles, sensible site structures, pages that loaded quickly and answered the question asked. Much was not. Since links were the signal, people manufactured links: networks of fake sites linking to each other, paid links, comment spam on blogs, directories created only to link out. Since Google rewarded pages that matched popular searches, companies produced pages for popular searches.

Google responded with constant changes to its ranking, some of them large enough to have names. An update in November 2003, known in the industry as Florida, sharply demoted many commercial sites that had relied on aggressive optimisation, weeks before Christmas. In February 2011 Google launched an update known as Panda, aimed at “content farms,” companies that used data on popular searches to commission large numbers of cheap articles written to rank for them, often carrying Google’s own AdSense ads. Google said the first Panda update affected about 11.8 percent of queries.¹⁸ Some of the largest content farms lost much of their traffic. In 2012 an update known as Penguin targeted manipulative links.

The result was an arms race with no end. Each signal Google used became a target. Each defence produced new methods of attack. Google kept its ranking factors largely secret, so that they would be harder to game, which also meant that businesses whose survival depended on search traffic could not know why they had risen or fallen. The company published the guidelines it gave to the human “quality raters” it employed to evaluate results, which emphasised expertise, authoritativeness and trustworthiness, with experience added in 2022. But the raters judged results; they did not set rankings, and the systems that did were known only in outline.

What those systems used was revealed partly in court. In the US antitrust trial in 2023, Google’s vice president of search, Pandu Nayak, testified about a system called Navboost that used data on what users clicked, collected over the previous thirteen months, to adjust rankings.¹⁹ In 2024 internal documentation describing thousands of attributes in Google’s search systems was published by mistake and confirmed as genuine by the company, though it did not show how each attribute was weighted.²⁰ Together they showed that the original idea, ranking by links, had long since been joined by ranking by behaviour: what millions of users did after they searched. That gave Google an advantage that grew with its size. The more searches it handled, the more it learned about which results satisfied people, and the better its results became.

From ten blue links to the answer

For its first decade, Google’s results page was a list of ten links with short extracts, the “ten blue links.” From 2007 the page began to change. Universal search mixed images, news, videos, maps and local businesses into the results. In May 2012 Google introduced the Knowledge Graph, a database of people, places and things and their relationships, built initially from sources including Freebase, Wikipedia and the CIA World Factbook. Announcing it, Google’s head of search, Amit Singhal, described the aim as understanding “things, not strings.”²¹ Results pages began to show boxes of facts about a person or a film, and “featured snippets” that quoted a passage answering the question at the top of the page.

These changes made Google better at answering simple questions. They also meant that more searches ended on Google. A user who asked for a film’s release date or the height of a mountain received the answer without visiting any site. A 2024 study by the analytics firm SparkToro, using clickstream data, estimated that 58.5 percent of Google searches in the United States and 59.7 percent in the European Union ended without a click to any website.²² Some of those searches were for facts that needed no click; some were searches whose answers had been extracted from sites that received nothing in return.

The space given to advertising also grew. In February 2016 Google stopped showing ads in the right-hand column on desktop and allowed up to four ads above the results for commercial queries.²³ The label distinguishing ads changed several times: a shaded background, then a small yellow “Ad” badge, then a green one, and from 2020 a bold black “Ad” in a format that resembled the small site icons now shown on organic results, a change Google partly reversed after criticism. In 2013 the FTC had written again to search companies, noting that compliance with its 2002 guidance had declined and that paid results had become less distinguishable from natural ones.¹⁵ The separation that had been the founding answer to the problem of mixed motives was still there. It had become less visible, and on many commercial searches the first organic result now sat below the fold of a phone screen.

The gatekeeper on trial

As Google came to handle about nine in ten searches worldwide, regulators began to ask whether its position had been won and kept on the merits.

The European Commission acted first. In June 2017 it fined Google €2.42 billion for favouring its own comparison shopping service: from 2008 Google had shown its own shopping results prominently, with pictures and prices, while rival comparison services appeared as ordinary blue links lower down, where few users clicked. In September 2024 the EU’s Court of Justice upheld the fine, holding that this unequal treatment by a dominant firm was not competition on the merits.²⁴ The EU also fined Google €4.34 billion in 2018 over conditions on Android phone makers and €1.49 billion in 2019 over restrictions in AdSense contracts. Its Digital Markets Act, applied from 2024, designated Google’s search as a “gatekeeper” service and banned self-preferencing outright. In July 2026 the Commission fined Google €460 million for continuing to favour its own services, such as shopping, hotels and transport, in search results, alongside a €430 million fine concerning its Play app store.²⁵

In the United States, the Department of Justice and a group of states sued Google in 2020. In August 2024 Judge Amit Mehta of the federal district court in Washington ruled that “Google is a monopolist, and it has acted as one to maintain its monopoly.” He found that Google held about 89.2 percent of the market for general search, rising to 94.9 percent on mobile devices, and that it had maintained that position through exclusive contracts that made it the default search engine on browsers and phones. Google paid $26.3 billion for such placements in 2021; its payments to Apple were estimated at $20 billion in 2022. The court also found that Google’s search was widely recognised as the best available. Users were not getting a bad product. The contracts were unlawful because they denied rivals the scale of queries and data needed to compete.²⁶ In remedies issued in September 2025 the court declined to order the sale of Chrome or Android, banned exclusive default contracts while allowing payments to continue, and ordered Google to share some search index and user-interaction data with qualified competitors, under a technical committee for six years. Both sides have appealed.²⁷

A second case concerned advertising technology. In April 2025 Judge Leonie Brinkema in Virginia found that Google had unlawfully monopolised the markets for software that publishers use to sell ad space and for the exchange on which that space is auctioned, and had unlawfully tied the two together. In September 2026 she declined to order Google to sell its ad exchange, and imposed behavioural remedies with a compliance monitor instead.²⁸ The European Commission separately fined Google €2.95 billion in September 2025 over its ad technology practices.

These cases did not find that Google’s ranking was corrupted by its advertising. They found something closer to what Brin and Page had described as “less blatant bias”: a gatekeeper using its control of the route to information to protect and extend its position, through defaults, through the placement of its own services and through owning both sides of an auction.

The machine that answers

The largest change to search since PageRank came from outside it. In late 2022 OpenAI released ChatGPT, and millions of people began asking a chatbot questions they would previously have typed into a search engine. The essay on large language models describes how those systems work and why they can be fluent and wrong. For search, the threat was simple. If a user received an answer in a conversation, they had no reason to look at a page of links, or at the ads beside them.

Google had developed much of the underlying technology, including the transformer architecture, but had been cautious about putting generated answers in front of users, partly because of the risk of errors and partly because answers threatened the ad-funded results page. In May 2024 it introduced AI Overviews in the United States: AI-generated summaries at the top of results, with links to sources. Within days, screenshots spread of overviews advising users to add glue to pizza sauce so the cheese would stick, and to eat a small rock a day. Google’s head of search, Liz Reid, explained that the errors had come from misinterpreted queries, satirical pages and joking forum posts, and from “data voids” where little reliable information existed, and the company narrowed the queries on which overviews appeared.²⁹ In May 2025 Google launched AI Mode, a conversational search interface, across the United States, and by the middle of that year said AI Overviews had more than two billion monthly users. OpenAI launched search inside ChatGPT in October 2024, and companies such as Perplexity built search engines around generated answers.

The new form changes the bargain on which the web was built. The old search engine sent readers to publishers, who earned money from those visits through ads and subscriptions, and so had a reason to keep writing the pages that the engine indexed. An answer engine reads those pages and summarises them, and fewer readers follow the links. A Pew Research Center study of the browsing of 900 American adults in March 2025 found that users who saw an AI summary clicked a traditional result in 8 percent of visits, compared with 15 percent for users who did not, and clicked a link within the summary itself in about 1 percent of visits.³⁰ In May 2025 Apple’s Eddy Cue testified that searches in Apple’s Safari browser had fallen in the previous month for the first time in over two decades, which he attributed to people turning to AI; Google responded that its overall queries, including from Apple devices, had grown.³¹ Both statements may be true. What is clear is that the answer is moving from the publisher’s page to the search engine’s, and the question of who pays the people who create the answers has been reopened.

What search is today

Search today has become several things at once. It is the web’s index: a continually updated copy of much of the public web, ranked by hundreds of signals including links, words, freshness, location and the recorded behaviour of billions of users. It is an answer service, increasingly written by AI, that resolves many questions without sending anyone elsewhere. It is the largest advertising market in the world, in which almost every commercial query triggers an auction among advertisers who pay for clicks. And it is a gatekeeper, regulated in Europe as one and found by American courts to have unlawfully maintained its monopoly.

Its basic interface has barely changed since 1998: an empty box that asks the user to say what they want. Everything behind the box has changed, and the box has become the most valuable piece of real estate in commerce.

How search changed the way we know and decide

Search changed what it means to know something, and how organisations and individuals decide. Several shifts stand out.

From remembering to retrieving

Before search, knowledge had to be held in memory, in a personal library or in the heads of colleagues. Search made it possible to look up almost anything in seconds, and people adjusted. A 2011 study by the psychologists Betsy Sparrow, Jenny Liu and Daniel Wegner found that people who expected to be able to look information up later remembered the information less well and remembered where to find it better.³² Knowing where to look became a substitute for knowing.

From browsing to asking

A library or a directory is browsed: the reader moves through a structure someone else designed and discovers things near what they were looking for. Search is asked: the reader types what they already think they want and receives the items that best match it. Search is better at answering questions and worse at suggesting the questions one did not think to ask.

From the expert’s judgement to the crowd’s

PageRank and its successors judged documents by what other people did with them, by who linked to them and who clicked on them. Authority on the web came to mean popularity among those who link and click, weighted by their own popularity. That was often a good proxy for quality. It was a different thing from the judgement of a cataloguer, an editor or an expert reviewer, and it rewarded those already found.

From reaching an audience to bidding for one

For businesses, search turned marketing into an auction. Instead of buying space in a newspaper and hoping the right people would see it, a company could bid to appear in front of people who had just typed the name of its product category. Advertising became measurable click by click, which made it far more efficient. It also meant that in many markets, being found came at a price set by competitors’ bids, paid to a single intermediary.

From the source to the summary

The latest shift, from links to generated answers, moves the reader one step further from the source. A list of links shows where an answer came from and invites comparison. A paragraph presents a synthesis whose sources are cited, if at all, in small print. The reader gains time and loses the chance to see who said what and to judge whether to believe them.

What the history suggests

Search repeats several patterns found elsewhere on this site. The first movers, AltaVista and the portals, did not see that the core task was the business. The company that invented the business model, GoTo, lost it to one that combined that model with a better product. The winner then defended its position with the methods of platforms everywhere: defaults bought from those who controlled distribution, its own services placed first and data that grew with scale.

But search also shows something more particular. The founders of Google identified the central problem, advertising’s pull on the ranking, more clearly than almost anyone, and designed an institution that held it at bay for a long time. The separation between organic results and ads, and the weighting of ads by usefulness to users, were real achievements. What weakened over time was not the wall between the two rankings so much as everything around it: the amount of the page given to ads, the visibility of the labels, the placement of Google’s own services and the shift from sending users away to keeping them. The bias Brin and Page feared did not arrive as a bought ranking. It arrived as a gradual change in the shape of the page.

Understanding the philosophy

The rest of this essay examines the ideas beneath web search, one at a time. Each is explained in plain terms, connected to the tradition of thought that has examined it, traced in the history above, tested against its strongest objection and related to how people and organisations decide today. The connections to philosophers, economists and sociologists are interpretations offered to make the ideas clearer. The engineers who built search engines were mostly solving practical problems, and did not, as a rule, set out to answer these thinkers.

1. How can you look for what you do not know?

The idea in plain terms

To search for something, you have to describe it. But often you search precisely because you do not know enough to describe it well. A person with a strange pain does not know the name of their condition; a manager entering a new market does not know which questions matter. The search box quietly assumes that the user already knows what they are looking for, at least well enough to put it into words. Every search engine is an attempt to work around that assumption.

The tradition

The puzzle is one of the oldest in Western philosophy. In Plato’s dialogue Meno, written in the early fourth century BCE, Socrates and Meno are trying to define virtue and keep failing. Frustrated, Meno raises an objection, now known as Meno’s paradox: how can you search for something when you do not know what it is? If you do not know it, you will not know what to look for, and even if you stumble on it, you will not recognise it as the thing you were seeking. And if you already know it, there is no need to search.

Socrates answers with a myth and a demonstration. The myth is that the soul is immortal and has known everything before birth, so that learning is a kind of recollection. The demonstration is a conversation with an enslaved boy who has never studied geometry. Through questions alone, Socrates leads him to see how to double the area of a square. The boy first answers confidently and wrongly, then becomes confused, and then finds the answer. Plato’s point, stripped of the myth, is that inquiry can begin from partial understanding and from recognising one’s own confusion. A person can know enough to recognise a good answer without knowing enough to state it in advance.

Later philosophers took the paradox in different directions. Aristotle distinguished between knowing something in general and knowing it in particular, so that one can know enough to search without knowing the answer. In the twentieth century the chemist and philosopher Michael Polanyi argued in The Tacit Dimension (1966) that Meno’s paradox shows we must possess a kind of knowledge we cannot fully articulate: we sense the presence of a problem and the direction of its solution before we can say what it is. The American pragmatist Charles Sanders Peirce called the creative leap from a puzzling observation to a candidate explanation “abduction,” and saw it as the starting point of all inquiry.

Information scientists met the same problem in practical form. In 1982 Nicholas Belkin and his colleagues described the “anomalous state of knowledge”: people approach an information system because they recognise a gap in what they know, but that gap is exactly what makes it hard for them to specify what would fill it. Their queries are therefore poor descriptions of their needs.

Where it shows up

Every stage of search history is an attempt to bridge this gap. The library catalogue offered subject headings, so that a reader who could not name a book could at least name its subject, if they could guess the heading. The directory let users browse, recognising what they wanted when they saw it, as Socrates’ pupil recognised the answer. The full-text engine allowed any word to be a starting point. Google’s ranking made a short, vague query produce useful results, because it relied on other people’s judgements about which pages were good. Spelling correction, query suggestions and autocomplete went further, offering users the words that other people had used for what seemed to be the same need. Conversational AI search is the latest attempt: users can describe their confusion in a sentence and refine it, as in a Socratic dialogue.

The strongest objection

One objection is that the paradox is a philosopher’s puzzle with little bearing on ordinary searching, since most searches are for things people can name perfectly well: a restaurant, a train time, a product. That is true of many queries. But the queries that matter most, the ones about health, law, money or a new field, are often the ones where users know least what to ask. Another objection is that modern search has largely solved the problem, because suggestions and AI answers help users move from a vague need to a clear question. They do, but by substituting other people’s questions, or a model’s guess, for the user’s own. The searcher who accepts the suggested query may never discover that they should have been asking something else.

What it means today

The search box rewards those who already know the right words. Experts search well; novices search badly, and are most likely to accept whatever comes first. For anyone making a decision in unfamiliar territory, the lesson of Meno applies directly. The first task is not to find the answer but to find the right question, and that usually requires reading around a subject, talking to people who know it and noticing one’s own confusion, before typing anything. An AI that turns a vague query into a confident answer can hide the very confusion that should have prompted further inquiry.

2. What makes a document relevant?

The idea in plain terms

A search engine’s whole purpose is to return relevant results. But relevance is not a property of a document alone. The same page can be exactly right for one person and useless for another who typed the same words: an expert who already knows the basics, a student who needs them, a buyer, a critic. Relevance is a relationship between a document, a person and a purpose, and a search engine can see only a small part of that relationship.

The tradition

Information science has argued about relevance for as long as it has existed. The Cranfield experiments in the 1960s treated it as something experts could judge in advance, by reading a document and deciding whether it answered a question. That made systems measurable, and it remains the basis of evaluation. But researchers soon noticed that judges disagreed, and that the same judge might change their mind. In 1973 the philosopher Patrick Wilson distinguished “logical” relevance, whether a document bears on a question, from what he called situational relevance: whether it bears on the concerns of a particular person in their particular situation, given what they already know. A document that repeats what the reader knows is logically relevant and situationally useless. In 1975 and in later work, the information scientist Tefko Saracevic surveyed the many meanings of relevance and argued that it is a human notion with several layers, from the algorithmic match between query and text to the cognitive, situational and emotional reasons a person finds something useful.

The deeper tradition here is phenomenology. The Austrian-American sociologist Alfred Schutz, writing in the 1940s and 1950s about how people make sense of everyday life, argued that what counts as relevant to a person depends on their projects and interests; each person lives within a “system of relevances” that determines what they notice and what they ignore. Relevance, for Schutz, is not found in the world. It is given by the purposes of the person looking at it.

Where it shows up

Search engines have spent decades trying to approximate the situational and personal side of relevance with signals they can measure. Spärck Jones’s weighting estimated which words carried meaning. PageRank estimated which pages the web’s authors considered important. Click data estimated which results satisfied people who typed similar queries. Location, language and search history personalised results. Each step brought the measured relevance closer to what people found useful, and each replaced the person’s own purposes with a statistical inference about people like them.

Advertising introduced a second notion of relevance. In the ad auction, an ad is relevant if users click on it and advertisers will pay for those clicks. Google’s weighting of bids by click-through rate was an attempt to make commercial relevance track user relevance, and it often does. But a click is not the same thing as satisfaction, and an ad can be relevant to a query while being a worse result than an unpaid page.

The strongest objection

The objection is pragmatic: whatever philosophers say, search engines work. Hundreds of millions of people find what they want every day, and statistical signals capture situational relevance well enough for most purposes. The gap Wilson and Schutz describe matters at the edges, for unusual needs, not at the centre. That is largely correct for common queries, which is why search has been so successful. But the edges are where expertise, minority viewpoints, new ideas and unusual situations live. A system that defines relevance by what satisfied most people who typed similar words will serve the typical user best and the atypical user worst.

What it means today

The results page answers the question “what did people like you find useful when they typed this?” That is a valuable question. It is not the same as “what do I need to know?” Decision-makers who rely on search, or on AI summaries built from search, should notice when their situation is unusual, because those are the situations in which a ranking tuned to the typical case will mislead. For anyone building a product with a ranking in it, relevance should be treated as a relationship to be investigated, not a property to be maximised.

3. The link as a vote: authority, citation and the crowd

The idea in plain terms

PageRank rested on a bold idea: that the judgements of millions of web authors, each deciding what to link to, could be added together into a measure of each page’s importance. No single author knew much about most pages. Together, the idea went, their choices contained more information about quality than any editor could supply. This is the idea that crowds can be wise, applied to authority.

The tradition

The idea has a long history. In 1785 the French philosopher and mathematician the Marquis de Condorcet proved what is now called the jury theorem: if each voter is more likely than not to be right on a yes-or-no question, and voters judge independently, then the probability that the majority is right approaches certainty as the group grows. In 1907 the statistician Francis Galton reported in Nature that the median of nearly 800 guesses at a livestock fair about the weight of an ox was within one percent of the true weight. In 1945 the economist Friedrich Hayek argued that market prices aggregate knowledge dispersed among millions of people, none of whom has it all, an idea explored in The Knowledge No One Has. The journalist James Surowiecki gathered such cases in The Wisdom of Crowds (2004).

Citation analysis applied a related idea to scholarship. Garfield treated citations as signals of a paper’s use, and Pinski and Narin weighted them by the standing of the citing journal. Brin and Page cited this tradition directly. But each of these traditions also identified the conditions under which aggregation fails. Condorcet’s theorem requires independence: if voters copy each other, the majority is no wiser than its leaders. Galton’s result depends on guesses made separately.

The sociologist Robert K. Merton described the most important failure in 1968. He noticed that eminent scientists received disproportionate credit for work done jointly with less famous colleagues, or independently by others, and named this the Matthew effect, after the verse in the Gospel of Matthew: “For unto every one that hath shall be given.” Recognition attracts more recognition. Cumulative advantage means that early, sometimes accidental, differences in reputation grow over time, independent of quality.

Where it shows up

PageRank was a near-perfect implementation of both the wisdom and the weakness. In 1998 links were largely independent judgements, which is why the ranking worked so well. But once Google became the main way people found pages, links stopped being independent. Authors linked to what they found, and they found what Google ranked highly. Pages at the top received more visitors, more visitors produced more links, and more links kept the pages at the top. In 2004 the computer scientists Junghoo Cho and Sourashis Roy studied this effect and found that search engines could entrench popular pages and make it harder for new high-quality pages to become known. Ranking by clicks intensified the loop: people mostly click on what is shown first. The survivorship of the already prominent is built into the signal.

At the same time, the value of being linked created a market in links, which destroyed the independence of the signal from the other side. The crowd whose judgement PageRank aggregated was gradually joined by people paid to impersonate it.

The strongest objection

The objection is that the alternatives are worse. Expert curation, as in Yahoo!’s directory, could not scale, and editors have their own biases. A crowd whose judgements are imperfectly independent is still better informed than any small group of experts. And Google does not simply count links; it devotes great effort to detecting manipulation and to bringing new pages to light. These points are fair. The question is not whether aggregated judgement is useful, which it plainly is, but whether those who rely on it understand its conditions: independence, diversity, and the absence of a feedback loop from the ranking back into the judgements.

What it means today

Any system that ranks by popularity also creates popularity. Product reviews, app store charts, bestseller lists, most-cited papers, and the internal dashboards companies use to choose which features to build all have the same structure. A decision-maker reading such a ranking should ask whether the judgements behind it were independent or whether people were following each other, and whether the ranking itself has been shaping what it measures. The top result may be the best. It may also simply be the one that was first.

4. When the ranking becomes the target

The idea in plain terms

Once a ranking decides who gets attention and money, people change their behaviour to rise in it. They do not necessarily become better at what the ranking was meant to measure. They become better at what the ranking actually measures. The signal then loses its value, and the ranker must find a new one, which starts the cycle again.

The tradition

The British economist Charles Goodhart observed in 1975 that any statistical regularity a central bank relied on for control tended to collapse once pressure was placed on it. The social psychologist Donald Campbell made a similar observation about social indicators in the 1970s: the more a quantitative indicator is used for social decision-making, the more it is subject to corruption pressures and the more it distorts the processes it was meant to monitor. The anthropologist Marilyn Strathern summarised the principle in the form now usually quoted: when a measure becomes a target, it ceases to be a good measure. The site’s idea page When a Measure Becomes a Target explores the principle in detail.

A related idea comes from the philosopher of science Ian Hacking, who described “looping effects” in the human sciences. When scientists classify people, the people classified can learn of the classification and change in response, so that the category and the people in it change each other over time. A ranking of web pages is a classification of this kind. The pages, and the people who write them, respond to how they are ranked.

Where it shows up

The history of search is an almost perfect case study. Keyword frequency was a good signal until authors stuffed keywords. Links were a good signal until links were bought and manufactured. Popular queries were a good guide to what people wanted to read until content farms wrote cheap pages to match them. Each Google update, Florida, Panda, Penguin and many more, was a move to restore the value of signals that had been gamed. The industry of search engine optimisation is a professional response to Goodhart’s law, and the secrecy of Google’s ranking factors is an institutional one: a measure that cannot be seen is harder to make into a target.

The looping effect also changed the web itself. Writers learned to put the answer in the first paragraph, to use the phrases people search for, to structure recipes around long introductions that held readers on the page with ads, to write headings as questions. Much of the web as it exists today has been shaped, often invisibly, by what ranked.

The strongest objection

One objection is that much optimisation is benign or useful. Pages that load quickly, answer clearly and are well organised are better for readers, and if Google rewarded those features, the web improved. The cycle of gaming and response is not a failure but the normal operation of a competitive system, which keeps improving as both sides adapt. That is partly right. Google has often chosen signals that align what ranks with what serves readers. But the alignment is never complete, and every gap is exploited at scale, because the rewards for ranking are so large.

What it means today

Any ranking with consequences will be gamed, and the more consequential it is, the more ingenious the gaming. Companies that set targets for their teams face the same problem in miniature. The response from search history is to keep the measure tied to what it stands for, to change it when it drifts, to look at what the measure misses and to be willing, as with Panda, to give up revenue to restore its meaning. A dashboard that stays the same for years is probably measuring how well people have learned to satisfy it.

5. The auction as a machine for telling the truth

The idea in plain terms

How much is a click from a person searching for “car insurance” worth? No one at Google knows. Each insurer knows roughly what such a customer is worth to it, but has every reason to pretend the figure is lower so as to pay less. The ad auction is a device for getting advertisers to reveal what attention is worth to them, and for allocating it to whoever values it most, without anyone having to trust anyone else.

The tradition

In 1961 the economist William Vickrey studied sealed-bid auctions and showed something surprising. If the highest bidder wins but pays the second-highest bid, every bidder’s best strategy is to bid exactly what the item is worth to them. Bidding more risks paying too much; bidding less risks losing an item one would have been glad to buy at the price. The auction makes honesty the best policy. Vickrey received the Nobel Prize in 1996.

His insight became part of a larger field called mechanism design, developed by Leonid Hurwicz, Eric Maskin and Roger Myerson, who shared the Nobel Prize in 2007. Mechanism design reverses the usual direction of economics. Instead of asking what outcome a given set of rules will produce, it asks what rules will produce a desired outcome when everyone acts in their own interest and keeps private what they know. Hurwicz called the central requirement “incentive compatibility”: the rules should make it in each participant’s interest to behave as the designer hopes.

The idea connects to an older one. Hayek argued that prices work as a communication system, transmitting knowledge that no planner could gather. An auction is a price-setting system in miniature, and a well-designed one turns private knowledge into a public number.

Where it shows up

Google’s auction is a practical descendant of Vickrey’s, though not a pure one. Charging the winner the next-highest bid, rather than its own, reduced the incentive to shade bids and the endless small adjustments that destabilised Overture’s first-price auctions. Weighting bids by expected clicks brought a second party into the mechanism: the user, whose clicks revealed which ads were useful. Edelman, Ostrovsky and Schwarz showed that the generalised second-price auction does not make honest bidding the best strategy in every case, but that it has a stable outcome yielding the same results as the version that does, and Varian reached similar conclusions independently. The design was found by engineering and explained by theory afterwards, which is a common sequence in mechanism design.

The auction made search advertising extraordinarily efficient. It set millions of prices every second without a sales force, it let the smallest business compete for attention on equal terms, and it gave Google a large share of the value that advertisers placed on each click.

The strongest objection

The objection is that an auction reveals willingness to pay, which is not the same as value to the user or to society. The advertiser who bids most for “debt relief” may be the one with the highest margins, not the one that helps the debtor most. Quality weighting corrects this only as far as clicks measure benefit, and clicks can be won by persuasive copy. Moreover, the auctioneer is not neutral. Google sets the reserve prices, decides how many ad slots appear, chooses how quality is calculated and, in display advertising, ran the exchange while also representing buyers and sellers, the conflict at the heart of the ad technology cases. A truth-telling mechanism is only as fair as the party that designs and operates it.

What it means today

Auctions are excellent at allocating scarce things to those who will pay most, and at revealing information people would otherwise conceal. They are not judgements of merit. Anyone whose customers find them through auctioned placements should understand that they are bidding against competitors in a market whose rules are set by the operator and can change. Anyone designing an internal allocation mechanism, for budgets, for headcount or for priorities, can learn from Vickrey: if the rules reward people for understating or overstating what they need, they will, and the right design makes honesty the winning move.

6. Attention is the scarce good, and someone pays for it

The idea in plain terms

Information on the web is abundant and mostly free. What is scarce is the attention of the people who might read it. A search engine allocates that attention, and its business model decides who pays for the allocation and therefore whose interests it serves. When users do not pay, the people who do are the ones who want users’ attention.

The tradition

In 1971 Herbert Simon, the economist and cognitive scientist who would receive the Nobel Prize in 1978, gave a lecture on designing organisations for an information-rich world. Information, he observed, consumes the attention of its recipients, so “a wealth of information creates a poverty of attention” and a need to allocate attention efficiently among the many sources that compete for it.³³ Simon’s point was that the bottleneck in an information-rich world is not information but the capacity to process it, an insight explored in the site’s idea on Bounded Rationality. In 1997 the physicist Michael Goldhaber argued that the internet economy would be an economy of attention, in which attention, not money or information, was the scarce resource people competed for.

Media scholars had made the commercial side of the point long before. In 1973 the artists Richard Serra and Carlota Fay Schoolman made a short video, Television Delivers People, whose scrolling text argued that commercial television’s product is its audience, sold to advertisers. The legal scholar Tim Wu traced the history of businesses that capture attention and resell it, from penny newspapers to Google, in The Attention Merchants (2016). Brin and Page’s appendix drew on the journalist Ben Bagdikian’s The Media Monopoly (1983) for their “historical experience with other media.”

Economists formalised the structure in the theory of two-sided markets. Jean-Charles Rochet and Jean Tirole, in work published in 2003, showed that platforms serving two groups, such as readers and advertisers or cardholders and merchants, typically charge one side little or nothing and recover their costs from the other, and that the choice of who pays shapes the whole platform. Tirole received the Nobel Prize in 2014, partly for this work.

Where it shows up

Search is a textbook two-sided market. Users pay nothing; advertisers pay for clicks. The early portals tried to sell attention by the thousand impressions and thereby had a reason to keep users on their pages. GoTo sold attention by the click and had a reason to put the highest bidder first. Google sold attention by the click, weighted by usefulness to users, and kept the paid ranking separate, which made its incentives better aligned with users than its rivals’. But the underlying structure was the one Brin and Page described: the paying customer is the advertiser. That structure shows in the growth of space given to ads, in the long pages that recipe sites built to hold readers in front of display ads and in the reluctance, for many years, to answer questions in a way that would end the visit. The site’s pattern who pays shapes what gets built collects other cases, and the email essay traces the same question for messages.

The strongest objection

The objection is that advertising made search free for everyone, including the billions of people who could never have paid for it, and that Google’s results remained good enough for it to win and keep users who were free to leave. If advertising had corrupted search badly, users would have switched. A subscription search engine would be available only to the rich. All of this has force. The advertising model is the reason universal search exists at all. But the Mehta ruling found that users’ freedom to leave was limited in practice by defaults, which complicates the claim that market choice disciplined the model. And the costs of the model are mostly subtle, appearing as changes in the shape of the page and the incentives of publishers, which users rarely notice and cannot easily compare.

What it means today

For every free service, it is worth asking who pays and what they are paying for, because the answer usually predicts how the service will change. Search engines funded by ads will tend to fill commercial queries with ads. AI answer engines will face the same choice, and the first signs of advertising in AI answers have already appeared. For businesses, attention bought through search is a cost that can rise without limit as competitors bid, and a dependence on a single intermediary. For readers, the useful habit is to notice which parts of any page are there because someone paid.

7. The gatekeeper and the politics of ranking

The idea in plain terms

Whoever controls what people see first has power over what they believe and buy. A search engine with nine in ten queries is not one voice among many. It is the gate through which most of the web is reached, and its choices about how to rank, what to show and which of its own services to place first have effects similar to those of a government’s choices, without the accountability.

The tradition

The idea of the gatekeeper comes from the social psychologist Kurt Lewin, who in 1947 studied how food reached American family tables and noticed that it passed through “channels” controlled by particular people, such as the person who did the shopping, whose choices determined what everyone ate. In 1950 the communication researcher David Manning White applied the idea to news, studying a wire editor he called “Mr. Gates” and showing how personal judgements decided which stories reached readers. Gatekeeping became a central concept in media studies.

In 2000 the philosophers Lucas Introna and Helen Nissenbaum published “Shaping the Web: Why the Politics of Search Engines Matters.” They argued that search engines were not neutral tools but made choices, technical and commercial, that systematically favoured some sites over others, typically large, popular and commercial ones, and that this mattered for a web that had been hoped to be a democratic medium. They called for disclosure of how search engines work and for public discussion of their values. In Algorithms of Oppression (2018), the information scholar Safiya Umoja Noble documented results that reproduced racial and gender stereotypes, arguing that commercial ranking systems reflect and amplify the biases in the data and the markets around them.

A separate tradition, in law, asks when private power over a market becomes a public concern. The American jurist Louis Brandeis warned in the early twentieth century against what he called the curse of bigness, arguing that concentrated economic power threatened democratic self-government as well as competition. From the late 1970s, American antitrust law was reshaped by the legal scholar Robert Bork’s argument that the law should protect consumer welfare, usually measured by prices, which made it hard to challenge firms offering free services. In 2017 the legal scholar Lina Khan, later chair of the FTC, argued in “Amazon’s Antitrust Paradox” that this standard missed the power of platforms that control the infrastructure on which their rivals depend.

Where it shows up

Google’s decisions about placement were consequential for whole industries. When it placed its own shopping results above rival comparison sites, their traffic fell; the EU’s courts held this to be an abuse. When it paid to be the default on browsers and phones, rivals could not reach the scale needed to improve; a US court held this to be unlawful monopoly maintenance. Its ranking updates could cut a company’s traffic overnight, with no appeal. The price-based antitrust standard struggled with a product that was free to users; Judge Mehta’s ruling focused instead on the advertising market and on rivals’ access to the scale that improves search quality, a sign of how the law has adapted.

The strongest objection

The objection is that Google earned its position by building a better product, which the court itself acknowledged, and that ranking is inherently a matter of judgement that no one can perform neutrally. Every search engine must decide what to show first, and calling those decisions “political” risks inviting governments to dictate them, which would be worse. Integrating specialised answers, such as maps and flights, made search better for users, even if it hurt intermediaries. These points are serious, and courts have been careful not to order Google to change its rankings directly. The answer of the gatekeeper tradition is not that ranking should be neutral, which is impossible, but that a gatekeeper of this importance should not use its position to favour itself, and should be open enough about its choices to be held to account.

What it means today

Any company that depends on a single gatekeeper, whether a search engine, an app store or a marketplace, holds its position at the gatekeeper’s discretion. The defensible strategy is to build direct relationships with customers wherever possible. For those who run platforms, the history of search is a warning that self-preferencing, once a platform becomes the market, eventually attracts regulation designed for it. And for anyone reading results, the gatekeeper’s choices are a reason to look past the first page when it matters.

8. Memory outside the head, and the end of the list

The idea in plain terms

Search has become part of how people think. We remember less and look up more, and treat the search engine as an extension of our own memory. That can be a great gain. But a memory that is outside the head is also controlled by someone else, and when the list of sources becomes a single generated answer, we no longer see where our knowledge came from.

The tradition

In 1998 the philosophers Andy Clark and David Chalmers published “The Extended Mind,” arguing that the mind does not stop at the skull. Their example was Otto, a man with memory loss who writes everything in a notebook he carries everywhere. When Otto wants to go to a museum, he consults the notebook for the address, as another person consults their memory. Clark and Chalmers argued that if an external resource is reliably available, readily used and automatically trusted, it plays the same role as biological memory and should count as part of the person’s mind. Psychologists had studied a related phenomenon, “transactive memory,” in which couples and teams divide remembering among their members; Daniel Wegner, one of the authors of the 2011 search study, developed that idea in the 1980s. That study found people treating the internet as a transactive memory partner.

Writers have long imagined a total store of knowledge and its strange effects. In the Argentine writer Jorge Luis Borges’s story “The Library of Babel” (1941), the universe is a library containing every possible book, and therefore every true statement, but also every false one and every meaningless one, so that the library’s inhabitants despair of ever finding anything. A complete store without a way of finding is no better than ignorance. Search is the answer to Borges’s library. It is also an answer that someone else designs.

The critique of outsourced thinking is old as well. In Plato’s Phaedrus, Socrates relates a story in which the Egyptian king Thamus warns that writing will produce forgetfulness and the appearance of wisdom without its substance, an argument examined in the idea of the pharmakon. In 2008 the writer Nicholas Carr revived the worry in an essay asking whether Google was making us stupid, arguing that constant searching and skimming eroded the capacity for sustained reading.

Where it shows up

The shift from the directory to the index to the answer box to the AI overview is a progressive extension of memory outside the head, and a progressive narrowing of what the user sees. The list of ten links still showed that there were several sources, written by different people, which could disagree. Featured snippets chose one. AI summaries synthesise many into a single voice. Clark and Chalmers’s conditions for an extended mind include that the resource is automatically trusted, which is exactly the condition that the errors of 2024 showed to be risky. And the publishers whose pages form the store are receiving fewer visits, which threatens the store itself: if the people who write the web stop being paid by the readers search sends them, they may stop writing, and the answers will be built from less.

The strongest objection

The objection is that extending memory has always made people more capable, not less. Writing, printing, libraries and encyclopedias each freed people from memorising and let them think about more. The Sparrow study shows people adapting sensibly, by remembering where to find things. AI answers that save time are a further step in the same direction, and users can always follow the links. This is a strong objection, and history largely supports it. But each earlier extension of memory kept the source visible: the book had an author, the encyclopedia had editors. An answer generated from many sources, by a system owned by the company that also sells the ads beside it, is a new kind of external memory, and its reliability and ownership are not yet settled.

What it means today

Using search as an external memory is reasonable, as long as one knows how reliable it is and who controls it. For decisions that matter, the old discipline still applies: go to the source, see who said it and when, and look for disagreement. For organisations, the knowledge their people “have” is increasingly knowledge they can retrieve, which is only as good as the retrieval system and the judgement of the person using it. And for anyone who publishes, the question of how the web’s writers will be paid when answers replace visits is now one of the most important open questions in the economics of information.

Bringing the threads together

Search began as an answer to Meno’s old question about how we can look for what we do not know, and it answered better than anything before it, by using the judgements of millions of other people. It defined relevance by what satisfied people like us, which serves the typical case and can fail the unusual one. It treated links as votes, which worked until the ranking began to shape the votes and the votes became a market. It made every signal it relied on into a target. It found, in the auction, a mechanism for pricing attention that made it one of the most efficient businesses ever built, and it lived with the founding tension between the user, who comes for the answer, and the advertiser, who pays for the user. As it became the gate through which most of the web is reached, its choices took on public weight, and courts on two continents found that it had used the gate to protect itself. Now, in becoming an answer engine, it is moving knowledge from the sources to a summary, and from the reader’s view to its own.

The auction behind the answer

Brin and Page warned in 1998 that advertising would bias search engines towards advertisers. They were right about the pressure and partly wrong about how it would appear. The company they founded built a careful separation between the ranking and the ads, and an auction that made usefulness to the user part of the price. For years that design delivered something remarkable: a free, fast and generally reliable route to much of the world’s published knowledge, funded by businesses that wanted to be found. Its failures came from other directions: from the arms race of people optimising for the ranking, from the gradual expansion of paid and self-serving space on the page and from the power that came with being the default way to find anything.

The history suggests a few working principles. A ranking is a set of choices, and whoever ranks for millions of people should be open about what those choices favour. Signals stay honest only while they are not the main route to reward, so any important measure must be watched and renewed. When the user does not pay, the business model must be designed to keep the payer’s interests from bending the product, and the design must be defended as the business grows. When a platform becomes the market, it cannot also compete inside it on unequal terms. And when a machine starts answering instead of pointing, it inherits responsibility both for the answer and for the people whose work it was built from. The search box asks each of us what we want to know. It is worth asking it, in return, how it decided what to show.

Selected bibliography

• Plato. Meno. Early fourth century BCE.
• Plato. Phaedrus. Fourth century BCE.
• Condorcet, Marquis de. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. 1785.
• Cutter, Charles A. Rules for a Printed Dictionary Catalogue. US Bureau of Education, 1876.
• Galton, Francis. “Vox Populi.” Nature 75 (1907): 450–451.
• Borges, Jorge Luis. “The Library of Babel.” 1941.
• Bush, Vannevar. “As We May Think.” The Atlantic Monthly, July 1945.
• Hayek, F. A. “The Use of Knowledge in Society.” American Economic Review 35, no. 4 (1945): 519–530.
• Lewin, Kurt. “Frontiers in Group Dynamics II.” Human Relations 1, no. 2 (1947): 143–153.
• White, David Manning. “The ‘Gate Keeper’: A Case Study in the Selection of News.” Journalism Quarterly 27 (1950): 383–390.
• Garfield, Eugene. “Citation Indexes for Science.” Science 122, no. 3159 (1955): 108–111.
• Vickrey, William. “Counterspeculation, Auctions, and Competitive Sealed Tenders.” Journal of Finance 16, no. 1 (1961): 8–37.
• Polanyi, Michael. The Tacit Dimension. Doubleday, 1966.
• Merton, Robert K. “The Matthew Effect in Science.” Science 159, no. 3810 (1968): 56–63.
• Simon, Herbert A. “Designing Organizations for an Information-Rich World.” In Computers, Communications, and the Public Interest, ed. Martin Greenberger. Johns Hopkins Press, 1971.
• Spärck Jones, Karen. “A Statistical Interpretation of Term Specificity and Its Application in Retrieval.” Journal of Documentation 28, no. 1 (1972): 11–21.
• Wilson, Patrick. “Situational Relevance.” Information Storage and Retrieval 9 (1973): 457–471.
• Hurwicz, Leonid. “The Design of Mechanisms for Resource Allocation.” American Economic Review 63, no. 2 (1973): 1–30.
• Saracevic, Tefko. “Relevance: A Review of and a Framework for the Thinking on the Notion in Information Science.” Journal of the American Society for Information Science 26 (1975): 321–343.
• Pinski, Gabriel, and Francis Narin. “Citation Influence for Journal Aggregates of Scientific Publications.” Information Processing & Management 12 (1976): 297–312.
• Bork, Robert H. The Antitrust Paradox. Basic Books, 1978.
• Belkin, N. J., R. N. Oddy and H. M. Brooks. “ASK for Information Retrieval.” Journal of Documentation 38 (1982): 61–71, 145–164.
• Bagdikian, Ben H. The Media Monopoly. Beacon Press, 1983.
• Hacking, Ian. “The Looping Effects of Human Kinds.” In Causal Cognition, ed. Dan Sperber et al. Oxford University Press, 1995.
• Goldhaber, Michael H. “The Attention Economy and the Net.” First Monday 2, no. 4 (1997).
• Brin, Sergey, and Lawrence Page. “The Anatomy of a Large-Scale Hypertextual Web Search Engine.” Computer Networks and ISDN Systems 30 (1998): 107–117.
• Clark, Andy, and David Chalmers. “The Extended Mind.” Analysis 58, no. 1 (1998): 7–19.
• Kleinberg, Jon M. “Authoritative Sources in a Hyperlinked Environment.” Journal of the ACM 46, no. 5 (1999): 604–632.
• Introna, Lucas D., and Helen Nissenbaum. “Shaping the Web: Why the Politics of Search Engines Matters.” The Information Society 16, no. 3 (2000): 169–185.
• Rochet, Jean-Charles, and Jean Tirole. “Platform Competition in Two-Sided Markets.” Journal of the European Economic Association 1, no. 4 (2003): 990–1029.
• Cho, Junghoo, and Sourashis Roy. “Impact of Search Engines on Page Popularity.” Proceedings of the 13th International World Wide Web Conference (2004).
• Battelle, John. The Search: How Google and Its Rivals Rewrote the Rules of Business and Transformed Our Culture. Portfolio, 2005.
• Edelman, Benjamin, Michael Ostrovsky and Michael Schwarz. “Internet Advertising and the Generalized Second-Price Auction.” American Economic Review 97, no. 1 (2007): 242–259.
• Varian, Hal R. “Position Auctions.” International Journal of Industrial Organization 25, no. 6 (2007): 1163–1178.
• Carr, Nicholas. “Is Google Making Us Stupid?” The Atlantic, July/August 2008.
• Levy, Steven. In the Plex: How Google Thinks, Works, and Shapes Our Lives. Simon & Schuster, 2011.
• Sparrow, Betsy, Jenny Liu and Daniel M. Wegner. “Google Effects on Memory.” Science 333, no. 6043 (2011): 776–778.
• Wu, Tim. The Attention Merchants. Knopf, 2016.
• Khan, Lina M. “Amazon’s Antitrust Paradox.” Yale Law Journal 126, no. 3 (2017): 710–805.
• Noble, Safiya Umoja. Algorithms of Oppression. NYU Press, 2018.
• United States v. Google LLC, No. 20-cv-3010 (D.D.C.), Memorandum Opinion, 5 August 2024.
• Court of Justice of the European Union. Google and Alphabet v Commission (Google Shopping), Case C-48/22 P, 10 September 2024.

Research limitations

The quotations from Brin and Page are taken from the text of their 1998 paper as published by Google Research. GoTo’s early bid prices and launch month come from secondary histories, which disagree on whether the service launched in 1997 or 1998; the February 1998 date for the bid-for-placement model is the more widely reported. The 2.7 million shares issued to Yahoo! settled both the patent dispute and a separate dispute over share warrants, as Google’s filings state. AdWords’ launch with about 350 advertisers and its move to cost-per-click in 2002 come from trade-press histories rather than Google’s own announcements; the attribution of the auction design to Eric Veach and Salar Kamangar relies on Steven Levy’s account. The 11.8 percent figure for Panda is Google’s own and is reported through secondary sources. Navboost is described from reports of trial testimony rather than the transcript. The figures for Google’s default payments are from the court’s findings as reported by the press; the Apple figure is an estimate in a partly redacted opinion. Search market shares vary by source and month; StatCounter’s figures, used here, measure page views and not queries. The SparkToro zero-click estimates come from clickstream panels and are approximations; sources disagree on its separate figures for clicks to the open web, which are not used here. Eddy Cue’s testimony about Safari searches was disputed by Google. The status of the US appeals and of the ad technology remedies is described as of early October 2026 from secondary reporting and may change. Gatekeeping, two-sided markets and the extended mind are applied to search as interpretation. The philosophical connections to Plato, Aristotle, Peirce, Polanyi, Belkin, Wilson, Saracevic, Schutz, Condorcet, Galton, Hayek, Merton, Goodhart, Campbell, Strathern, Hacking, Vickrey, Hurwicz, Simon, Goldhaber, Serra and Schoolman, Wu, Rochet and Tirole, Lewin, White, Introna and Nissenbaum, Noble, Brandeis, Bork, Khan, Clark and Chalmers, Wegner, Borges and Carr are the author’s interpretation; their works are paraphrased apart from short, attributed phrases. The decisions in the decisions box are judged by the author, separating what could be known at the time (process) from what happened (call).

Footnotes

1. Alphabet Inc., “Alphabet Announces Fourth Quarter and Fiscal Year 2025 Results,” 4 February 2026: total revenues of $402,836 million for 2025 and Google Search & other revenues of $63,073 million in the fourth quarter. ↩

2. Sergey Brin and Lawrence Page, “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” Computer Networks and ISDN Systems 30 (1998), including Appendix A, “Advertising and Mixed Motives”; PDF at Google Research. ↩ ↩²

3. Charles A. Cutter, Rules for a Printed Dictionary Catalogue (Washington: US Bureau of Education, 1876), statement of “objects”; see Wikipedia, “Charles Ammi Cutter”. ↩

4. Vannevar Bush, “As We May Think,” The Atlantic Monthly, July 1945. ↩

5. Karen Spärck Jones, “A Statistical Interpretation of Term Specificity and Its Application in Retrieval,” Journal of Documentation 28, no. 1 (1972); see also her later account, “IDF term weighting and IR research lessons” (2004). ↩

6. Eugene Garfield, “Citation Indexes for Science,” Science 122 (1955); on its relation to PageRank, Stephen J. Bensman, “Eugene Garfield, Francis Narin, and PageRank,” arXiv:1312.3872. ↩

7. “The Search Is On: 20 Years of Internet Search,” PCWorld; Wikipedia, “Alan Emtage”. ↩

8. Wikipedia, “AltaVista”. ↩

9. Wikipedia, “PageRank”, on RankDex and HITS; Jon M. Kleinberg, “Authoritative Sources in a Hyperlinked Environment,” Journal of the ACM 46 (1999), first presented in January 1998. ↩

10. “GoTo: The Forgotten Search Engine,” The History of the Web; John Battelle, The Search (2005), excerpt; Acquired, “Overture”. ↩

11. Google Inc., Form 10-Q for the quarter ended 30 September 2004, on the settlement with Yahoo! of the Overture patent suit (US Patent 6,269,361) and the warrant dispute. ↩ ↩²

12. “2002 In Review: Google Powers AOL; AdWords Go Cost-Per-Click,” Search Engine Land; Wikipedia, “Google Ads”. ↩

13. Hal Varian, “More insights about Quality Score and Ad Rank,” Google Ads blog, June 2014; Hal R. Varian, “Position Auctions,” International Journal of Industrial Organization 25 (2007). ↩

14. Benjamin Edelman, Michael Ostrovsky and Michael Schwarz, “Internet Advertising and the Generalized Second-Price Auction,” American Economic Review 97, no. 1 (2007). ↩

15. Federal Trade Commission, letter to Commercial Alert, 27 June 2002; on the 2013 letter, eWeek, June 2013. ↩ ↩²

16. “Google buys Applied Semantics,” InfoWorld, April 2003. ↩

17. “Yahoo buys Overture in $1.63 billion deal,” Information Age, July 2003; on the 2009 agreement with Microsoft, socaltech. ↩

18. “The most memorable Google algorithm updates of all time,” Search Engine Land. ↩

19. On Pandu Nayak’s testimony, see “Google’s Navboost algorithm”, which summarises the trial record. ↩

20. Wikipedia, “2024 Google Search documentation leak”. ↩

21. Amit Singhal, “Introducing the Knowledge Graph: things, not strings,” Google blog, 16 May 2012. ↩

22. Rand Fishkin, “2024 Zero-Click Search Study,” SparkToro, July 2024, using Datos clickstream data. ↩

23. “Google says goodbye to right-side ads,” CMSWire, February 2016; on the 2020 redesign, 9to5Google. ↩

24. Court of Justice of the European Union, press release 135/24, 10 September 2024, on Case C-48/22 P. ↩

25. European Commission, press release IP/26/1670, 23 July 2026; Tech Policy Press. ↩

26. United States v. Google LLC, Memorandum Opinion (D.D.C., 5 August 2024), as reported by NPR via GPB and MobileSyrup. ↩

27. “What the Google antitrust remedies ruling means,” Cato Institute, September 2025; on the appeal, MacRumors, May 2026. ↩

28. “Google Won’t Have To Break Up Its Ad Tech Business, Judge Brinkema Rules,” AdExchanger; Goodwin, “Antitrust & Competition Technology Q3 2026 Update”; on the EU fine, BNN Bloomberg. ↩

29. Liz Reid, “AI Overviews: About last week,” Google blog, 30 May 2024. ↩

30. Pew Research Center, “Google users are less likely to click on links when an AI summary appears in the results,” 22 July 2025. ↩

31. “Searches in Safari dipped for the first time,” Yahoo Tech, May 2025; Google’s response reported by MacTrast. ↩

32. Betsy Sparrow, Jenny Liu and Daniel M. Wegner, “Google Effects on Memory,” Science 333 (2011); Columbia University news release. ↩

33. Herbert A. Simon, “Designing Organizations for an Information-Rich World,” in Martin Greenberger, ed., Computers, Communications, and the Public Interest (1971); quoted in The Conversable Economist. ↩