Every team that runs HTTP services on Kubernetes eventually hits the same mystery: the app handles SIGTERM correctly, readiness probes are configured, the rolling update strategy looks textbook — and deploys still produce a small burst of 502s or connection resets. The code is fine. The mental model is wrong. Kubernetes never promises to stop sending traffic to a pod before it starts killing it. Those are two separate processes, they run in parallel, and nothing synchronizes them.

What actually happens when a pod is deleted

When you delete a pod (or a rolling update does it for you), the API server marks the pod for deletion and the kubelet starts a shutdown sequence bounded by terminationGracePeriodSeconds, which defaults to 30 seconds:

  1. If the container defines a preStop hook, the kubelet runs it first. Per the docs, hooks "are not executed asynchronously from the signal to stop the Container; the hook must complete its execution before the TERM signal can be sent."
  2. The kubelet sends SIGTERM to the container's main process.
  3. The kubelet waits out the remainder of the grace period.
  4. If the process is still alive when the grace period expires, it gets SIGKILL.

During all of this, kubectl get pods shows Terminating — which, a detail worth knowing, is a kubectl display field, not an actual pod phase. The pod's phase remains Running until the containers actually exit.

So far, so reasonable. The problem is what happens at the same time.

The race: two tracks, no coordination

Deleting a pod kicks off a second, independent chain of events: the endpoints machinery notices the pod is going away and removes its IP from the Service's endpoints. That update then has to propagate to every consumer of endpoint data:

  • kube-proxy on every node watches endpoint changes and rewrites its iptables (or IPVS) rules to stop routing to the dying pod.
  • Ingress controllers typically watch endpoints too — most route traffic directly to pod IPs, skipping the Service's virtual IP entirely — and must reload or reprogram their backends.
  • Anything else consuming endpoints (service meshes, custom load balancers, DNS-based discovery) has its own propagation delay.

As the learnkube deep-dive on graceful shutdown puts it: "Kubernetes is still routing traffic to the IP address, but the Pod is no longer there. The Ingress controller, kube-proxy, CoreDNS, etc., didn't have enough time to remove the IP address from their internal state." And on timing: "there is no guarantee on how long it will take to remove the IP address from their internal state. For some, it could take less than a second; for others, it could take more."

Meanwhile the kubelet is not waiting for any of this. SIGTERM is delivered on the kubelet's schedule, not after the data plane has converged. The failure mode follows directly:

  1. Kubelet sends SIGTERM; your well-behaved server stops accepting new connections and starts draining.
  2. A node whose iptables rules haven't been updated yet routes a fresh request to the pod.
  3. Connection refused. The client sees a 502 from the ingress, or a reset.

The better your app's SIGTERM handling — the faster it closes its listener — the wider this window is. Correct application code makes the symptom worse.

The fix is deliberately dumb: sleep

You cannot make endpoint propagation synchronous. What you can do is delay SIGTERM until propagation has (probably) finished, and the preStop hook is the lever, because the kubelet must run it to completion before it's allowed to send the signal:

yaml
containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "15"]

During those 15 seconds the pod is unmarked from endpoints, kube-proxy and the ingress controller catch up, new traffic stops arriving — and then the process gets SIGTERM and drains whatever is genuinely in flight. The learnkube article suggests 15 seconds as a starting point within the default 30-second grace period, and increasing it ("perhaps to 20 or 25 seconds") if some component in your cluster takes longer to drop the endpoint. The application needs no changes and doesn't even know the sleep happened.

The classic annoyance was that exec-based sleep requires a sleep binary in the image — awkward for FROM scratch and distroless images. That's exactly the motivation given in KEP-3960, which added a native sleep action to the kubelet itself: alpha in v1.29, beta in v1.30, stable in v1.34.

yaml
lifecycle:
  preStop:
    sleep:
      seconds: 15

Get the arithmetic right

The subtle trap: the grace period countdown starts before the preStop hook runs, not after. The lifecycle hooks documentation is explicit: "The Pod's termination grace period countdown begins before the PreStop hook is executed, so regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period."

The docs give the worked example: with terminationGracePeriodSeconds: 60, a hook that takes 55 seconds, and an app that needs 10 seconds to shut down cleanly, the total is 65 seconds — the container is SIGKILLed before it finishes. The budget is one shared pot:

terminationGracePeriodSeconds  ≥  preStop duration + worst-case drain time

With the default 30-second grace period and a 15-second sleep, your app has 15 seconds to drain after SIGTERM. If long requests or connection draining can take longer, raise terminationGracePeriodSeconds — it's a per-pod-spec field, and bumping it costs nothing when shutdown is fast, since the kubelet proceeds as soon as the process exits.

Two more sharp edges from the same docs page:

  • Hook delivery is at-least-once. "A hook may be called multiple times for any given event" — rare (e.g., a kubelet restart mid-delivery), but it means a preStop that does real work, not just sleeping, must be idempotent.
  • A failed or hanging hook doesn't save you. If the hook hangs, the pod just sits in Terminating until the grace period expires and the kill proceeds. A failed preStop kills the container and leaves a FailedPreStopHook event in kubectl describe pod — worth checking when shutdown behaves strangely, because nothing louder will tell you.

Where sleeping stops working

The sleep trick assumes connections are short. For WebSockets, long polling, or jobs that hold a connection for hours, no reasonable grace period covers the drain, and cranking terminationGracePeriodSeconds to 6 hours means every deploy and node drain stalls behind zombie pods. For that class of workload the learnkube article describes "rainbow deployments": create a new Deployment per release, shift new connections to it, and delete the old Deployment only when its connections have actually drained — moving the drain out of pod termination entirely.

For ordinary request/response services, though, the checklist is short:

  1. Handle SIGTERM: stop accepting, drain in-flight, exit.
  2. Add a preStop sleep (native sleep action on v1.30+, exec sleep otherwise) so endpoint propagation wins the race.
  3. Size terminationGracePeriodSeconds to cover the sleep plus your drain, because they share one budget.

The deeper lesson generalizes past Kubernetes: in any distributed system, "stop sending traffic here" is an eventually-consistent broadcast, not a transaction. Readiness signals, load balancer deregistration, DNS TTLs — every one has a propagation window, and the boring, robust engineering move is the same everywhere: keep serving correctly for a little while after you've been told to die.

References

kubernetesdeploymentssigtermnetworkingreliability