Set a CPU limit on a container and the kernel does not give you "2 cores." It gives you a budget: so many microseconds of CPU time per accounting period, enforced by CFS bandwidth control. When the budget runs out mid-period, every thread in the cgroup is frozen until the next period starts — even if the machine is otherwise idle. That freeze is throttling, and the engineering story behind it includes a worked example that surprises most people, a quota-distribution mechanism almost nobody reads about, and a kernel bug that made mostly-idle services miss their limits for over a year.

The budget, not the cores

Two cgroup values define the limit: cpu.cfs_quota_us (the budget) and cpu.cfs_period_us (the window, default 100 ms). Quota defaults to -1, meaning unconstrained. A "2 CPU" limit is really quota = 200 ms per 100 ms period; the kernel enforces a minimum of 1 ms for quota and period and a maximum period of 1 s.

The part that bites: quota is spendable anywhere in the period, at any parallelism. Indeed's engineering team, who ended up fixing the kernel bug below, worked through the canonical example in their write-up. Give an application 0.4 CPU — 40 ms of quota per 100 ms period — and hand it a request that needs 200 ms of CPU. It burns 40 ms, sits throttled for 60 ms, and repeats. The request completes after 440 ms of wall time spanning five periods, four of them throttled, for 240 ms of accumulated throttled_time. The CPU was free the whole time. Latency came entirely from the limit.

You can watch this happening in the cgroup's cpu.stat:

nr_periods     5
nr_throttled   4
throttled_time 240000000   # nanoseconds

nr_throttled / nr_periods is the throttled percentage, and it is the first thing to graph when a limited service has mysterious tail latency. For a genuinely CPU-bound group, the kernel documentation notes nr_periods should roughly equal nr_throttled — being throttled every period is what "at its limit" looks like.

Quota travels in 5 ms slices

Here is the mechanism the cpu.stat numbers hide. Quota is not a single counter that threads decrement. It lives in a global per-cgroup pool, refilled to the full quota at each period boundary, and it is handed out to per-CPU runqueues in slices as threads become runnable on each CPU. The slice size is /proc/sys/kernel/sched_cfs_bandwidth_slice_us, default 5 ms. Bigger slices mean fewer trips to the global pool (less lock contention); smaller slices mean finer-grained accounting.

So a 20-thread application on a big node is constantly checking out 5 ms slices onto whichever CPUs its threads land on. When all threads on a CPU go to sleep, the runqueue returns its local runtime to the global pool — all except 1 ms, a compile-time constant called min_cfs_rq_runtime, kept local to avoid hammering the global lock.

That design raises an obvious question: what happens to checked-out runtime that nobody used when the period ends? The answer changed three times, and the middle answer was the bug.

Five years broken, one year painful

In v3.16-rc1 (2014), commit 51f2176d74ac introduced a conditional that extended each CPU's local deadline by a tick — with the effect that local slices never actually expired. Nobody noticed, because unused leftovers quietly carrying over is mostly harmless.

In the 4.18 kernel, commit 512ac999d275 ("sched/fair: Fix bandwidth timer clock drift condition") fixed a real problem — clock skew between cores was causing premature quota expiry — and, as a side effect, made expiration work as originally designed. That commit was backported widely, landing in RHEL, CentOS, and Ubuntu distribution kernels.

Suddenly, expiration was real, and a whole class of workloads started getting throttled while visibly under their limits. The failure shape: a highly threaded, non-CPU-bound application — exactly what a typical Kubernetes service is — spreads threads across many CPUs, each runqueue checks out a slice, each thread does a little work and blocks. Stranded local runtime now expired at the period boundary instead of being usable, so the group was throttled in a high percentage of periods while consuming a fraction of its quota. Kubernetes issue 67577 collected the real-world reports.

Indeed hit it in production with an application allocated 4.1 CPUs that never used more than about 0.5 CPU — mostly idle, heavily throttled, with worst-case response latency over two seconds. Dave Chiluk at Indeed wrote a minimal reproducer, fibtest: fast threads computing Fibonacci iterations flat-out, slow threads doing 100 iterations then sleeping 10 ms, each pinned to its own CPU — a caricature of asynchronous worker threads that briefly wake and block.

The fix, commit de53fd7aedb1 ("sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices," merged in 2019 via Peter Zijlstra and backported to 4.19 stable that November), is notable for what it deletes: rather than repairing expiration, it removes the mechanism entirely — runtime_expires, expires_seq, expire_cfs_rq_runtime, gone. 65 insertions, 83 deletions across three files. On an 80-CPU machine running fibtest with 10 ms of quota per 100 ms period, the commit reports almost a 30x performance improvement with quota still enforced. Indeed's two-second worst-case latency dropped to 30 ms.

The tradeoff the kernel now documents

Removing expiration means a group can slightly exceed its quota in a given period: runtime already sitting on a lightly loaded runqueue stays spendable past the boundary. The commit bounds the overrun by the leftover per-CPU runtime — typically no more than min_cfs_rq_runtime (1 ms) per CPU — and the current sched-bwc documentation carries the same caveat: a highly threaded group "can briefly exceed its quota," while average usage over longer windows still honors the limit. CPU-bound groups see no difference; they drain their slices every period anyway.

That philosophical concession — strict per-period enforcement is less useful than letting unused budget carry — was later made explicit as a feature. cpu.cfs_burst_us (default 0, and capped at the quota) lets a group deliberately accumulate unused runtime and spend it in spikes, with nr_bursts and burst_time in cpu.stat tracking how often that happens. The kernel docs frame it statistically: provision for typical demand, say p(95), instead of worst-case p(100), and let burst absorb the gap — their example puts two such tasks both staying within quota 90.25% of the time and colliding only 0.25% of the time.

What to actually do

  • Graph nr_throttled / nr_periods and throttled_time for any cgroup with a limit. CPU-usage graphs will look fine while throttling destroys your tail latency — usage averaged over a minute hides a 60 ms freeze inside every 100 ms.
  • Know your kernel. The painful window is 4.18 (or a distro kernel that backported 512ac999d275) up to wherever de53fd7aedb1 landed in your stable tree; it reached 4.19 stable in November 2019. On modern kernels, the five-year-broken, one-year-painful expiry behavior is simply gone.
  • Remember quota is concurrency-multiplied. Twenty threads can drain a 2-CPU quota in 10 ms of wall time and then freeze for 90. The period, not the core count, sets your worst-case stall.
  • Consider burst before raising limits. If your workload is spiky but light on average, cpu.cfs_burst_us is the mechanism designed for exactly that shape — bounded carryover instead of permanently overprovisioned quota.

CPU limits aren't wrong — they are the only thing standing between a runaway neighbor and your latency SLO on a shared node. But they are a budget with a 100 ms accounting cycle and a 5 ms distribution grain, not a smaller computer. Most "we set limits and everything got slow" stories are one of these mechanisms, and cpu.stat will tell you which.

References

cfs-bandwidthcpu-throttlingkubernetescgroupslinux-kernel