Two write() Calls, One 40 ms Stall: How Nagle's Algorithm and Delayed ACKs Deadlock Until a Kernel Timer Fires

Two algorithms, written by different people in the early 1980s to solve different problems, each work fine alone. Put one on each end of a TCP connection and a request/response that should take one round trip stalls until a kernel timer fires — 40 ms minimum on Linux, up to 500 ms by spec. Neither side is broken. Each is politely waiting for the other.

1984: the 4000% overhead problem

RFC 896, written by John Nagle at Ford Aerospace and dated January 6, 1984, describes what it calls the small-packet problem. A telnet session sends one keystroke per TCP segment: 1 byte of data under 40 bytes of IP and TCP headers, a 41-byte packet that the RFC describes as 4000% overhead. On a congested network those tinygrams get dropped, retransmitted, and can degrade a connection until it aborts.

The fix Nagle proposed is one sentence, and the striking thing about it is what it doesn't contain:

inhibit the sending of new TCP segments when new outgoing data arrives from the user if any previously transmitted data on the connection remains unacknowledged

No timers. The RFC is explicit that the inhibition is unconditional — no timers, no size tests. Small writes coalesce in the send buffer while one segment is in flight, and the arrival of the ACK releases whatever has accumulated. The algorithm is self-clocking: on a fast network ACKs come back quickly and small segments flow almost unimpeded; on a slow network, data batches up into fewer, fuller packets exactly when batching helps most.

The RFC's worked example is a telnet user typing every 200 ms over a 5-second-RTT link: with no control, 25 packets in flight and 4000% overhead; with a send-timer scheme, 1500% overhead; with Nagle's rule, 24 characters coalesce into one packet and overhead drops to 320%. For bulk transfer the cost is negligible — the RFC's file-transfer example loses 1.6% (254 seconds vs. 250), and it claims the scheme keeps "at least 98%" of baseline throughput.

As a solo act, it's an elegant algorithm. The problem is that it assumes ACKs arrive as fast as the network allows.

Berkeley's bet: the delayed ACK

Around the same time, BSD implemented delayed acknowledgments. Instead of ACKing every segment immediately, the receiver holds the ACK briefly, hoping to piggyback it on response data or to ACK two segments with one packet. RFC 1122 (1989) standardized the behavior:

A TCP SHOULD implement a delayed ACK.
The delay MUST be less than 0.5 seconds.
In a stream of full-sized segments there SHOULD be an ACK for at least every second segment.

Note what the heuristic is really doing. In Nagle's own words (from a Hacker News comment he wrote as user Animats): "A delayed ACK is a bet that the other end will reply to what you just sent almost immediately." For telnet with remote echo, the bet pays off — the echo rides back with the ACK. For a modern RPC server that reads a complete request before writing anything, the bet loses, and the loss is paid in timer milliseconds.

On Linux the timer bounds are in include/net/tcp.h:

#if HZ >= 100
#define TCP_DELACK_MIN  ((unsigned)(HZ/25))  /* minimal time to delay before sending an ACK */
...
#define TCP_DELACK_MAX  ((unsigned)(HZ/5))   /* maximal time to delay before sending an ACK */

HZ/25 is 40 ms; HZ/5 is 200 ms. That 40 ms floor is the number you keep seeing in latency histograms.

The deadlock: write, write, read

Neither algorithm stalls on its own. The stall needs both, plus an application that issues two writes before reading — say, a client that writes a request header, then writes the body, then blocks on the response:

Client writes the header. No data is in flight, so the segment goes out immediately.
Client writes the body. It's smaller than one MSS and the header is still unacknowledged, so Nagle holds it.
The server receives the header. It's an incomplete request, so the application doesn't respond yet. The TCP stack has no data to piggyback on, and one small segment isn't the two full-sized segments that force an ACK — so the delayed-ACK timer starts.
Nagle waits for the ACK. The ACK waits for the timer. Nothing moves.
The timer fires (≥ 40 ms on Linux, up to 500 ms per RFC 1122), the bare ACK goes out, Nagle releases the body, the server finally has a complete request.

A request that should cost one round trip — about 500 μs within a datacenter, per Marc Brooker — costs 40 ms instead. That's an 80x penalty, invisible to the application, applied per stall. Pipelined protocols hit it repeatedly.

Brooker's post on this is titled "It's always TCP_NODELAY. Every damn time," and he means it operationally: when debugging tail latency in a distributed system, the first thing he checks is whether TCP_NODELAY is set, because it so often isn't.

Nagle himself has been blunt about where the fault lies: "The real problem is not tinygram prevention. It's ACK delays, and that stupid fixed timer... I did tinygram prevention (the Nagle algorithm) and Berkeley did delayed ACKs, both in the early 1980s... The combination of the two is awful." And: "That still irks me."

What to actually do

Set TCP_NODELAY. Per tcp(7): "If set, disable the Nagle algorithm." RFC 1122 makes the opt-out a MUST — every conforming stack has to let an application disable Nagle per connection — which is why the option exists everywhere:

int one = 1;
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one));

This is why so much infrastructure ships with it on by default, and Brooker's recommendation is to enable it "without guilt" for latency-sensitive systems on modern hardware. The 1984 trade no longer binds: headers are a rounding error on gigabit links, and RFC 896 itself was reclassified Historic (obsoleted by RFC 7805).

Don't recreate the tinygram problem in userspace. With Nagle off, a loop that calls write() per byte really will emit a packet per byte. The fix is the application's job: buffer and batch your writes (writev, a buffered writer) so each syscall hands the kernel a complete message.

Know TCP_CORK, but for the right job. Linux's cork ("don't send out partial frames") holds partial segments until you uncork — useful for prepending a header to a sendfile() payload. It has its own 200 ms ceiling, after which queued data is sent anyway, and it combines with TCP_NODELAY only since Linux 2.5.71. It's a batching tool, not a latency tool.

Don't bet on TCP_QUICKACK. It attacks the other side of the deadlock by disabling delayed ACKs, but tcp(7) is explicit that the flag "is not permanent, it only enables a switch to or from quickack mode" — the kernel re-enters delayed-ACK mode on its own. Brooker's advice is not to rely on it: it's non-portable, its semantics are odd, and it does nothing about your own kernel holding your own writes.

The durable lesson is bigger than TCP. Nagle's rule and the delayed ACK are both local heuristics, each betting on the other endpoint's behavior — one bets ACKs come back fast, the other bets data is about to follow. Each bet is sound in isolation; composed, they wait on each other. When two timers or two batching heuristics face each other across a protocol boundary, check what each one is assuming the other will do — that assumption is where the 40 ms hides.

References

tcpnagledelayed-acklatencynetworking