End-to-End Arguments in System Design

The situation

By 1981 the ARPANET had been running for more than a decade, and the internet’s protocols, TCP and IP, were being finalised. Network designers faced a recurring question. Many functions could be placed in the network itself or in the computers attached to it: checking that data arrived intact, retransmitting lost packets, encrypting messages, acknowledging delivery. Building them into the network seemed helpful, since every application would benefit automatically.

Jerome Saltzer, David Reed and David Clark, researchers at MIT, noticed that engineers had long used a particular argument against that approach, “with neither explicit recognition nor much conviction,” as they put it. They set out to state it precisely.

What it argues

The paper opens with a claim about the designer’s craft: “Choosing the proper boundaries between functions is perhaps the primary activity of the computer system designer.” It then states the argument in a form that has been quoted ever since:

“The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the end points of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible. (Sometimes an incomplete version of the function provided by the communication system may be useful as a performance enhancement.)”

The example is “careful file transfer”. To move a file from one computer to another without damage, many things can go wrong: reading from disk, bugs in the software, failures in the network, writing to the other disk. A perfectly reliable network would guard against only one of these. The only way to be sure the file arrived intact is for the application at the receiving end to check it, for example with a checksum, and ask for it again if necessary. Since that end-to-end check is needed anyway, building elaborate reliability into the network adds cost without removing the need for the check. It can still help performance, by reducing how often the end-to-end check fails, but it cannot replace it.

The authors apply the same reasoning to encryption, duplicate message suppression, delivery acknowledgement and other functions, and suggest that it applies to operating systems and file systems too. Low-level mechanisms for these functions, they conclude, “are justified only as performance enhancements.”

The idea underneath

At its heart the end-to-end argument is about knowledge: a function belongs where the knowledge needed to perform it correctly resides. That makes it a close cousin of Hayek’s argument about dispersed knowledge. The network, like a central planner, cannot know what each application needs. The applications at the edges can.

It also has a strategic dimension the authors hinted at and later writers developed. A simple, general network that does not try to anticipate applications leaves room for applications nobody has imagined. Email, the web, streaming video and voice calls all ran over the same simple internet without it needing to be redesigned for each. Keeping the core simple preserved options, which connects the argument to the value of irreversibility and flexibility. Lawrence Lessig and others later used the argument to defend what became known as network neutrality.

How it was received and what followed

The paper became one of the most cited in computer networking and a defining statement of the internet’s design philosophy. The email essay and the communication tools essay show its consequences: a network that delivered anything to anyone without judging it, which enabled extraordinary growth and also spam, fraud and abuse.

By 2001 David Clark, one of the authors, and Marjory Blumenthal were asking how the principle should adapt to a world of untrustworthy endpoints, commercial pressures and governments wanting control. Firewalls, content delivery networks, filtering and other functions placed in the network have since become common. The argument has not been refuted, but it has been tested by a world very different from the research network in which it was formulated.

Reading it today as a decision-maker

The argument generalises well beyond networks. For any platform, ask where each function’s necessary knowledge resides. If only the customer’s application can do something correctly, building it into the platform duplicates effort and may constrain them. If the platform has knowledge no customer has, it belongs in the platform.

The strategic lesson is about generality. Platforms that stay simple and general support uses their designers never imagined. Platforms that build in assumptions about their uses may serve those uses better, at the cost of foreclosing others. That is a permissive defaults trade, and its costs, the abuse that open systems attract, are real too.

Sources and limitations

Quotations are from the authors’ own copy of the paper hosted at MIT, a revised version of the 1981 conference paper as published in 1984, checked against the extracted text. The later debate is summarised from Blumenthal and Clark’s 2001 paper and Lessig’s book. The connection to Hayek is the author’s interpretation.