The Universal Machine and the End of the Free Lunch

Before it named a machine, “computer” named a job. For centuries, and well into the twentieth, computers were people, often teams of women, who worked through long calculations by hand or with mechanical calculators: navigation tables, ballistics, astronomical predictions, census totals. A difficult problem was solved by dividing it among many people working in parallel.

Today the word names something that is partly an industry and partly a utility. When a person asks an AI system a question, the answer is produced by specialised chips in a data centre that may be hundreds of kilometres away. The International Energy Agency estimated that data centres used around 415 terawatt-hours of electricity in 2024, about 1.5 percent of the world’s consumption, and projected that this would more than double to around 945 TWh by 2030, with artificial intelligence as the most important driver.¹

Between those two images lies the history of compute systems. It is usually told as a story of ever-faster machines. A more revealing way to tell it is as a long bet on one idea, the universal machine, and what happened when the physics that paid for that bet stopped paying.

Machines for particular jobs

Before the 1940s, calculating machines were built for particular jobs. Hollerith’s tabulators sorted and counted census cards. Mechanical differential analysers solved certain kinds of equations by turning shafts and wheels. During the Second World War, codebreaking machines were built to attack specific ciphers. Each machine embodied its task in its physical structure. To do a different job, you built, or substantially rebuilt, a different machine.

The universal machine

The idea that would dissolve this arrangement came first from mathematics. In 1936 Alan Turing, working on a problem in logic, described an abstract machine that reads and writes symbols on a tape according to a table of rules. He then showed something remarkable: a single “universal” machine of this kind, given a description of any other such machine on its tape, could imitate it. In principle, one machine was enough for every computation.

Engineering arrived at a practical version of the same idea within a decade. ENIAC, completed at the University of Pennsylvania in 1945, was general-purpose in what it could calculate, but it was programmed by physically connecting cables and setting switches, so changing the task meant rearranging the machine. The same group, led by J. Presper Eckert and John Mauchly, was already designing a successor. In June 1945 John von Neumann wrote a report on that successor, the EDVAC, which described a computer whose instructions were stored in its memory alongside its data.² The credit for the stored-program idea has long been disputed; the report circulated under von Neumann’s name, while Eckert, Mauchly and others had contributed to the design. What is not disputed is the idea’s significance. Once a program is data, changing what a machine does means writing new data, not rewiring it.

This is the philosophical foundation of modern computing. A computer is not a machine for a task. It is a machine for any task, defined by the program it happens to be running. Everything that followed, from operating systems to app stores, assumes that generality.

Physics pays for generality

General-purpose machines were initially slow and expensive compared with special-purpose ones. What made generality win was a sequence of discoveries that made general-purpose hardware cheaper and faster at an astonishing rate.

The transistor, invented at Bell Labs in 1947, replaced fragile vacuum tubes with solid-state switches. In 1958 and 1959 Jack Kilby and Robert Noyce independently showed how to build many transistors and their connections on a single piece of semiconductor: the integrated circuit.

In April 1965 Gordon Moore, then at Fairchild Semiconductor, published a short article in Electronics magazine. His observation was economic as much as technical: “The complexity for minimum component costs has increased at a rate of roughly a factor of two per year.” He expected the rate to continue for at least a decade, so that “by 1975, the number of components per integrated circuit for minimum cost will be 65 000.”³ Moore later revised the pace to a doubling about every two years, and the trend, eventually called Moore’s law, held for decades.

A second principle explained why more transistors did not simply mean hotter chips. In 1974 Robert Dennard and colleagues at IBM described how shrinking a transistor’s dimensions, along with its voltage, could keep the power used per unit of chip area roughly constant. Smaller transistors switched faster and used less power each. This “Dennard scaling” meant that each new generation of chips could run faster without melting.

Together these two trends produced something unusual in the history of technology: a product that improved dramatically every year without its users having to do anything. In 1971 Intel’s 4004 put a complete general-purpose processor on a single chip; it had been developed for a Japanese calculator company, which needed logic for its machines, and a programmable processor turned out to be more practical than custom circuits. That small decision captured the era’s logic. Why design a special chip for each job when a general one, programmed differently, would do, and would get faster every year?

The age of the free lunch

For roughly three decades, the answer to almost every performance problem was to wait. Software written for one generation of processors ran faster on the next without being changed. Specialised hardware struggled to compete, because by the time a custom chip was designed and manufactured, general-purpose processors had often caught up.

The industry’s architecture followed the economics. Computing moved from shared mainframes to minicomputers to personal computers, because cheap general-purpose chips made it possible to put a capable machine on every desk. Architects refined how processors executed instructions, with ideas such as reduced instruction set computing in the 1980s, which simplified instructions so they could be executed faster. But the overall direction was settled. General-purpose processors, made faster by physics, ran almost everything.

The wall

The free lunch ended early in the 2000s, and the clearest announcement came from a software engineer. In an article published in Dr. Dobb’s Journal in March 2005, Herb Sutter wrote: “CPU performance growth as we have known it hit a wall two years ago.” Around the beginning of 2003, he showed, the steady climb in processor clock speeds had sharply flattened, for “not just one but several physical issues, notably heat (too much of it and too hard to dissipate), power consumption (too high), and current leakage problems.” Chip makers, he wrote, “have run out of room with most of their traditional approaches to boosting CPU performance,” and were turning instead to multiple cores on a single chip.⁴

The reason was that Dennard scaling had broken down. As transistors became extremely small, voltages could no longer be reduced in proportion, and current leaked even when transistors were off. Packing more of them more densely now meant more heat per square millimetre. Moore’s law continued for a while to provide more transistors, but they could no longer all be run faster.

Sutter’s title stated the consequence: “The Free Lunch Is Over.” If processors would no longer make existing programs faster automatically, programmers would have to restructure their software to do many things at once. The cost of performance, which physics had paid for decades, moved onto the people writing software. Parallel programming, long a specialist discipline, became everyone’s problem.

The return of the utility

As individual processors stopped getting much faster, computing grew outward instead. Large internet companies built enormous facilities containing tens of thousands of servers and treated each building as one machine; Luiz André Barroso and Urs Hölzle of Google described this in a 2009 book titled The Datacenter as a Computer. In 2006 Amazon began renting computing capacity by the hour, and cloud computing made such warehouse-scale machines available to anyone with a credit card.

This was, in one sense, a return to the past. The time-sharing pioneers of the 1960s had imagined computing as a utility supplied like electricity. The personal computer decentralised computing for a few decades; the cloud recentralised it. The economics had changed: when one large shared system could be run far more efficiently than millions of small ones, computing flowed back toward the centre.

Machines for particular jobs, again

The deepest change came in the hardware itself. If general-purpose processors could no longer get faster, then the remaining route to large gains was to build hardware suited to particular kinds of work, the strategy that the universal machine had once made obsolete.

The most important example came from an unexpected direction. Graphics processors had been designed to draw images for games, which required performing the same arithmetic on millions of pixels at once. In 2006 and 2007 NVIDIA introduced CUDA, which let programmers use these massively parallel chips for general calculation. In 2012 a team at the University of Toronto trained a neural network for image recognition, later known as AlexNet, that dramatically outperformed earlier methods; its paper reports that training took “five to six days on two NVIDIA GTX 580 3GB GPUs.”⁵ Neural networks, it turned out, consisted mostly of exactly the kind of parallel arithmetic that graphics chips were built for. A specialised chip designed for one purpose had found its true calling in another.

Google went further and designed a chip only for neural networks. A 2017 paper describing its Tensor Processing Unit, deployed in its data centres from 2015, began with a statement of the new philosophy: “Many architects believe that major improvements in cost-energy-performance must now come from domain-specific hardware.” For the neural network workloads it was built for, the TPU was “on average about 15X - 30X faster than its contemporary GPU or CPU, with TOPS/Watt about 30X - 80X higher.”⁶

The same year, John Hennessy and David Patterson, whose textbooks had taught generations of engineers computer architecture, received the Turing Award. In their award lecture, published in Communications of the ACM in 2019 as “A New Golden Age for Computer Architecture,” they argued that the end of Dennard scaling and the slowing of Moore’s law were not merely problems but an opening for domain-specific architectures, paired with domain-specific languages, to deliver the gains general-purpose processors no longer could.⁷

The pendulum had swung back. After fifty years of making one machine do everything, the most important computing hardware of the new era was again being designed for particular jobs.

Compute as a resource

The specialised chips arrived at the moment when a new kind of software needed them in unprecedented quantities. Epoch AI, a research group that tracks the resources used to build AI systems, estimates that the training compute used for notable AI models has grown by roughly four to five times every year since 2010, with most of that growth coming from increased spending rather than from better hardware alone.⁸ No exponential in the history of computing had grown that fast, including Moore’s law.

This changed what compute is in economic and political terms. It became a resource to be acquired in bulk, measured in thousands of specialised chips and in megawatts of power. Governments began treating advanced chips and the equipment to make them as strategic goods, subject to export controls. And the energy cost of computation, long invisible to users, became a public question. The IEA’s figures place data centres alongside major industries in their electricity use, with the United States accounting for 45 percent of data centre consumption in 2024, China 25 percent and Europe 15 percent.¹

Understanding the philosophy

The history of compute is usually told as engineering, but each turn in it rests on an idea about what a machine is, who should pay for its improvement, and where its power should live. What follows takes those ideas one at a time. Each is explained in plain terms, connected to the tradition of thought that has examined it most closely, traced in the history above, tested against its strongest objection, and related to computing today. The links to philosophers and economists are interpretations offered to make the ideas clearer; the engineers in this history did not, as a rule, cite them.

1. A machine defined by what it is told, not by what it is

The idea in plain terms

Before the stored program, a machine’s purpose was built into its body. After it, a machine’s purpose was whatever its program said. The same physical object could be a calculator, a typewriter, a game or a scientific instrument, depending on the data placed in its memory.

The tradition

Philosophers have long distinguished what a thing is from what it does. Aristotle defined many things by their function: an eye is what sees, a knife is what cuts. The stored-program computer breaks this habit. Its function is not fixed by its structure at all; it is supplied from outside, as information. In the seventeenth century Gottfried Wilhelm Leibniz dreamed of a universal symbolic language and a calculating machine that could settle any reasoned dispute by computation. Turing’s 1936 universal machine was a mathematical realisation of part of that dream: one machine, any procedure.

Where it shows up

ENIAC had to be physically rearranged to change its task; the EDVAC report described a machine whose instructions were stored as data.² Intel’s 4004 began as logic for a calculator, and became a general processor because programming one chip proved more practical than designing many. Every later layer of computing, from operating systems to app stores, assumes a machine whose purpose is software.

The strongest objection

No real machine is purely universal. Every processor has a particular memory size, speed and instruction set, and some tasks run on it far better than others. Universality is a statement about what is possible given unlimited time and memory, not about what is sensible in a working day.

What it means today

The tension between “can do anything” and “does some things well” is the thread of the whole history. Software still treats computers as universal. Hardware, increasingly, does not.

2. Generalists, specialists and the size of the market

The idea in plain terms

A general-purpose machine is a generalist; an accelerator is a specialist. Which one wins depends less on which is cleverer than on how much of one kind of work there is to do.

The tradition

Adam Smith opened The Wealth of Nations (1776) with the division of labour, using the example of a pin factory in which workers who each perform one small operation produce vastly more than any worker making whole pins alone. The title of his third chapter added the condition that matters here: “That the Division of Labour is Limited by the Extent of the Market.”⁹ Specialisation pays only when there is enough demand for the specialised task to keep the specialist busy. A village cannot support a pin factory; a nation can.

Where it shows up

For decades, the market for any single kind of computation was too small to justify custom chips that would be overtaken by the next general-purpose generation within a few years. Graphics was the first workload large enough, through the game industry, to sustain a specialised processor. Neural networks then became the largest single workload in computing, and Google’s TPU paper stated the new conclusion: “major improvements in cost-energy-performance must now come from domain-specific hardware.”⁶ Smith’s condition had been met. The market for one kind of arithmetic had become large enough to pay for machines that do nothing else.

The strongest objection

Specialists are brittle. A chip designed around today’s neural network architectures may fit tomorrow’s poorly, and a generalist adapts where a specialist must be replaced. This is why accelerators are still surrounded by general-purpose processors, and why the most successful ones, such as GPUs, kept enough programmability to follow the field as it changed.

What it means today

Every specialised chip is a bet that a kind of work will stay large and stable for long enough to repay its design. When the bet is right, as with matrix arithmetic for AI, the gains are enormous. When it is wrong, the hardware becomes an expensive curiosity.

3. A prediction that organised its own fulfilment

The idea in plain terms

Moore’s law was not a law of nature. It was an observation about economics that an industry then treated as a target, planning its research, investment and products so that the next doubling would arrive on time.

The tradition

In 1948 the sociologist Robert K. Merton described the self-fulfilling prophecy: a belief about the future that, by changing how people act, makes itself come true. The idea belongs to a wider question in the philosophy of technology, the debate over technological determinism, which asks whether technology develops according to its own internal logic and shapes society, or whether social choices shape technology.

Where it shows up

Moore’s 1965 article was explicitly about the cost of components, and it projected a trend for “at least 10 years.”³ It held for decades partly because chip makers, equipment suppliers and their customers coordinated around it, publishing shared industry roadmaps that set targets for each generation. The “law” worked as a promise the industry kept to itself.

The strongest objection

Coordination cannot overrule physics. The end of Dennard scaling around 2003 showed the limit of a self-fulfilling prophecy: belief and investment could keep making transistors smaller, but they could not stop small transistors from leaking current and producing heat.⁴ At that point the determinist reading looks partly right after all. Material constraints set the boundaries within which social choices operate.

What it means today

Exponential trends in computing, including the rapid growth of AI training compute, are often discussed as if they were natural laws. The history of Moore’s law suggests a more careful reading: such trends are sustained by decisions and money, and they continue until they meet a constraint that decisions and money cannot move.

4. When the free lunch ends, someone pays

The idea in plain terms

For thirty years software got faster without anyone doing anything, because hardware improved underneath it. When that stopped, the cost of performance did not disappear. It moved to programmers, chip designers and, ultimately, to the electricity grid.

The tradition

Economists use the word externality for a cost that falls on someone who did not choose it, a concept developed by Arthur Cecil Pigou in The Economics of Welfare (1920). A factory that pollutes a river imposes a cost on people downstream that does not appear in the factory’s prices. The phrase “there is no such thing as a free lunch,” popularised by economists in the twentieth century, makes the same point in ordinary language: every apparent gift is paid for somewhere.

Where it shows up

Herb Sutter’s 2005 article named the moment the payment fell due: if processors would no longer speed up existing programs, programmers would have to restructure them for concurrency.⁴ Hennessy and Patterson argued that the next gains would require domain-specific hardware and languages, a cost paid in engineering effort.⁷ And the energy cost of computation, long invisible to users, now appears in national electricity statistics.¹

The strongest objection

Not every cost that moved is an externality in the strict sense. Programmers who learned parallel programming were paid to do so, and data centres pay for their electricity. What has changed is that the cost became visible and had to be chosen, rather than being absorbed silently by the physics of shrinking transistors.

What it means today

Questions about the energy and water used by data centres are, at bottom, questions about who bears the costs of computation and whether its prices reflect them. Those questions did not need asking while each generation of chips did more with less.

5. Information is physical

The idea in plain terms

Software feels weightless. But every calculation is a physical event: electrons moving, switches changing state, energy turning into heat. There is no computation without matter and energy.

The tradition

In 1867 the physicist James Clerk Maxwell imagined a tiny being, later called Maxwell’s demon, that could sort fast and slow molecules and so appear to defeat the second law of thermodynamics. Resolving the puzzle took most of a century, and the resolution ran through information. Rolf Landauer argued in 1961 that erasing a bit of information has an unavoidable minimum energy cost, and he later summed up the view in the title of a 1991 article: “Information is Physical.”¹⁰ Knowledge itself, on this account, has a thermodynamic price.

Where it shows up

Every turning point in this history was a physical limit. Vacuum tubes failed and burned power; transistors replaced them. Dennard scaling broke because of heat and leakage. The largest AI systems are now planned around power supply, cooling and grid connections. The abstractions of computing, programs and data, run on matter that warms the world.

The strongest objection

Landauer’s limit is far below the energy real chips use, by many orders of magnitude, so it does not explain present-day costs. Most of today’s energy goes into moving data and into the inefficiencies of real circuits, not into the irreducible cost of erasing bits.

What it means today

The objection points in a hopeful direction: because current computing operates so far above the theoretical floor, large improvements in efficiency remain physically possible. But the principle stands. Computation is a use of the physical world, and its scale now makes that impossible to ignore.

6. Where the power of computation lives

The idea in plain terms

Computing has moved from shared central machines, to personal machines, and back to central machines in the cloud. Each shift changed who controls computation and who can use it.

The tradition

In a 1964 essay, “Authoritarian and Democratic Technics,” the historian Lewis Mumford argued that technologies come in two broad kinds: large, centralised systems that concentrate power, and small, distributed tools that individuals and communities can control. He worried that the first kind tended to absorb the second. The question of whether a technology’s architecture carries political consequences has remained central to the philosophy of technology since.

Where it shows up

The mainframe era concentrated computing in a few institutions; the personal computer distributed it to individuals; the cloud and the data centre, described by Barroso and Hölzle as a single warehouse-scale computer, concentrated it again. The demand of frontier AI for vast numbers of specialised chips and large amounts of power has intensified that concentration, since few organisations can assemble such resources.

The strongest objection

Centralisation can also distribute access. Cloud computing lets a student or a small company rent, for an afternoon, hardware they could never afford to buy. Concentration of ownership is not the same as concentration of use, and the cloud arguably made powerful computing more widely available than personal ownership ever did.

What it means today

Both things are true at once: more people can use powerful computation than ever before, and fewer organisations control the largest systems. Whether capable AI runs mainly in distant data centres or increasingly on personal devices is, in Mumford’s terms, a question about the political shape of computing.

7. General methods, specialised machines

The idea in plain terms

At the level of software, AI has moved toward ever more general methods that learn from data and computation rather than relying on knowledge built in by hand. At the level of hardware, those same methods have pushed computing toward highly specialised chips. Generality in one layer has produced specialisation in the layer beneath it.

The tradition

Philosophy has argued for centuries about whether knowledge is mostly built in or mostly learned. Rationalists such as Descartes and Leibniz held that the mind comes equipped with innate ideas; empiricists such as John Locke described the mind at birth as something like a blank page, written on by experience. In 2019 the AI researcher Richard Sutton gave the debate a computational form in a short essay, “The Bitter Lesson”: “The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.”¹¹

Where it shows up

AlexNet’s success in 2012 came from a general learning method given far more computation, on graphics cards, than earlier approaches had used.⁵ The growth of training compute since then, around four to five times a year by Epoch AI’s estimate, is the bitter lesson pursued as an industrial strategy.⁸ To supply that computation efficiently, hardware specialised: GPUs tuned for neural networks, TPUs built for nothing else.

The strongest objection

Scaling has limits of its own: the supply of high-quality data, the cost of energy and chips, and the diminishing returns that eventually affect any single method. Much current research works in the opposite direction, seeking smaller and more efficient models that do more with less computation. The bitter lesson describes a long historical pattern, not a guarantee about the future.

What it means today

The universal machine has not disappeared; it has moved. Turing’s idea that one general mechanism could do anything now lives in learning algorithms, while the hardware beneath them has become more specialised than at any time since the 1940s. The dream of generality persists, but its price is paid in specialised silicon and electricity.

Bringing the threads together

The history of compute is a long negotiation between generality and its price. The stored program made a machine’s purpose a matter of information, and for half a century physics made general machines improve so fast that specialisation could not compete. Moore’s law worked partly as a prophecy that organised its own fulfilment, until heat and leakage refused to cooperate. When the free lunch ended, the cost of performance moved to programmers, to chip designers and to the energy system, and the size of the AI market finally made Adam Smith’s condition for specialisation true for silicon. Along the way, computing reminded us that information is physical, and that where computation lives is a question about power in both senses of the word.

What a computer is for

Return to the question at the start. Computing returned to building machines for particular jobs because the conditions that made one machine for every job the best choice were temporary. They depended on a remarkable, decades-long run of physical improvements that let general-purpose hardware outpace everything else. When the run slowed, the old logic of specialisation reasserted itself, now applied to the kinds of work, especially neural networks, that matter most at present.

The universal machine has not been abandoned. Every specialised accelerator is still controlled by general-purpose processors running general-purpose software, and Turing’s insight still defines what a computer is. What has changed is that universality now has a visible price, paid in heat, in energy and in the concentration of resources. The people who once worked as computers divided a hard problem among many hands. Today’s computers divide it among many chips in buildings that draw power on the scale of cities. The question of what a computer is for, once answered by whatever program it happened to be running, is increasingly answered in watts.

Selected bibliography

• Turing, A. M. “On Computable Numbers, with an Application to the Entscheidungsproblem.” Proceedings of the London Mathematical Society s2-42 (1936–37): 230–265.
• von Neumann, John. First Draft of a Report on the EDVAC. Moore School of Electrical Engineering, University of Pennsylvania, 30 June 1945. archive.org
• Moore, Gordon E. “Cramming More Components onto Integrated Circuits.” Electronics 38, no. 8 (19 April 1965); reprinted in Proceedings of the IEEE 86, no. 1 (1998).
• Dennard, Robert H., et al. “Design of Ion-Implanted MOSFETs with Very Small Physical Dimensions.” IEEE Journal of Solid-State Circuits 9, no. 5 (1974).
• Landauer, Rolf. “Irreversibility and Heat Generation in the Computing Process.” IBM Journal of Research and Development 5, no. 3 (1961).
• Sutter, Herb. “The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software.” Dr. Dobb’s Journal 30, no. 3 (March 2005). gotw.ca
• Barroso, Luiz André, and Urs Hölzle. The Datacenter as a Computer. Morgan & Claypool, 2009.
• Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. “ImageNet Classification with Deep Convolutional Neural Networks.” NeurIPS (2012).
• Jouppi, Norman P., et al. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” ISCA (2017). arXiv:1704.04760
• Hennessy, John L., and David A. Patterson. “A New Golden Age for Computer Architecture.” Communications of the ACM 62, no. 2 (2019): 48–60.
• Epoch AI. “Training compute of notable models since 2010.” Data insight. epoch.ai
• International Energy Agency. Energy and AI. April 2025. iea.org
• Smith, Adam. An Inquiry into the Nature and Causes of the Wealth of Nations. 1776. Book I, chapters 1–3.
• Merton, Robert K. “The Self-Fulfilling Prophecy.” The Antioch Review 8, no. 2 (1948): 193–210.
• Pigou, Arthur Cecil. The Economics of Welfare. Macmillan, 1920.
• Landauer, Rolf. “Information is Physical.” Physics Today 44, no. 5 (1991): 23–29.
• Mumford, Lewis. “Authoritarian and Democratic Technics.” Technology and Culture 5, no. 1 (1964): 1–8.
• Sutton, Richard S. “The Bitter Lesson.” 13 March 2019. incompleteideas.net

Research limitations

The EDVAC report and the 1974 Dennard paper are described from their well-established contents rather than quoted; credit for the stored-program concept remains disputed among historians, and the essay presents it as such. Hennessy and Patterson’s 2019 article is paraphrased from its published abstract and widely reported argument rather than quoted. Epoch AI’s growth estimates have been revised as its dataset expanded, ranging from about 4.2 to 4.7 times per year in different versions. The IEA figures are estimates and projections with acknowledged uncertainty, especially beyond 2030. The interpretation of computing history as a pendulum between universal and specialised machines, governed by the economics of scaling, is the author’s. The philosophical section’s connections to Aristotle, Leibniz, Smith, Merton, Pigou, Maxwell, Mumford, Locke and the rationalists are the author’s interpretation; Merton, Pigou and Mumford are paraphrased rather than quoted, and only Sutton’s essay and the TPU paper speak directly about computing hardware and methods.

Footnotes

1. International Energy Agency, Energy and AI (April 2025), executive summary; figures as reported in coverage including DatacenterDynamics and Scientific American. ↩ ↩² ↩³

2. John von Neumann, First Draft of a Report on the EDVAC (30 June 1945); on its circulation and the dispute over credit, see Thomas Haigh, Mark Priestley and Crispin Rope, ENIAC in Action (MIT Press, 2016). ↩ ↩²

3. Gordon E. Moore, “Cramming More Components onto Integrated Circuits,” Electronics 38, no. 8 (1965), as reprinted in Proceedings of the IEEE 86, no. 1 (1998), p. 83. ↩ ↩²

4. Herb Sutter, “The Free Lunch Is Over,” Dr. Dobb’s Journal 30, no. 3 (March 2005), section “Obstacles, and Why You Don’t Have 10GHz Today.” ↩ ↩² ↩³

5. Alex Krizhevsky, Ilya Sutskever and Geoffrey E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,” Advances in Neural Information Processing Systems 25 (2012). ↩ ↩²

6. Norman P. Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit,” Proceedings of the 44th International Symposium on Computer Architecture (2017), abstract. ↩ ↩²

7. John L. Hennessy and David A. Patterson, “A New Golden Age for Computer Architecture,” Communications of the ACM 62, no. 2 (2019), based on their 2018 Turing Award lecture; see the lecture abstract at iscaconf.org. ↩ ↩²

8. Epoch AI, “Training compute of notable models since 2010,” data insight, epoch.ai/data-insights/compute-trend-post-2010. ↩ ↩²

9. Adam Smith, The Wealth of Nations (1776), Book I, title of chapter 3. ↩

10. Rolf Landauer, “Irreversibility and Heat Generation in the Computing Process,” IBM Journal of Research and Development 5, no. 3 (1961); Rolf Landauer, “Information is Physical,” Physics Today 44, no. 5 (1991). ↩

11. Richard S. Sutton, “The Bitter Lesson,” 13 March 2019, first paragraph. ↩