Kubernetes CPU limits look like a ceiling on average usage: set limits.cpu: "4" and your container gets up to four cores. That mental model is wrong in a way that produces one of the most confusing production signatures in container platforms: a pod reported at a fraction of its limit that is simultaneously being throttled hard. Indeed's engineering team documented an application allocated 4.1 CPUs, consuming roughly 0.5, and still throttled badly enough to correlate directly with multi-second tail latency.

To see why, you have to look at what a CPU limit actually compiles down to.

A CPU limit is a bank account with a 100 ms statement cycle

The kubelet enforces CPU limits through the Linux kernel's CFS bandwidth control (--cpu-cfs-quota defaults to true). The Kubernetes docs are blunt about the mechanism:

"cpu limits are enforced by CPU throttling. When a container approaches its cpu limit, the kernel will restrict access to the CPU corresponding to the container's limit. Thus, a cpu limit is a hard limit the kernel enforces."

CFS bandwidth control has exactly two knobs per cgroup, and the kernel documentation defines the contract in one sentence: "Within each given period (microseconds), a task group is allocated up to quota microseconds of CPU time." The defaults are cpu.cfs_period_us = 100ms and cpu.cfs_quota_us = -1 (unconstrained). Setting a limit turns the quota on: a limit of 1 CPU becomes 100 ms of CPU time per 100 ms period; a limit of 4 becomes 400 ms per period. (The period is configurable via the kubelet's --cpu-cfs-quota-period, but the kernel default is 100 ms.)

Notice what this is not: it is not a rate limit on average consumption over any human-visible timescale. It is a budget that refills every tenth of a second. Everything surprising about CPU throttling follows from that window size.

Throttling is a full stop, not a slowdown

When a cgroup's quota for the current period is gone, the kernel doesn't slow its threads down — it parks them. Per the kernel docs, throttled threads are "not able to run again until the next period when the quota is replenished." Every runnable thread in the container freezes until the period boundary.

Two pieces of arithmetic show why this hurts latency-sensitive services specifically.

Quota is consumed by all threads in parallel. A container limited to 1 CPU gets 100 ms of budget per period, but that budget is shared across every core its threads run on. If a request fans out across 8 threads that all run simultaneously, the budget can be gone in 12.5 ms of wall-clock time — followed by up to 87.5 ms in which the container cannot execute a single instruction. A modern runtime with a pool of worker threads, a concurrent GC, and a few background loops burns quota far faster than "one CPU's worth per second of wall clock" intuition suggests.

Throttling converts CPU time into latency cliffs. A request that needs 80 ms of CPU under a 50 ms-per-period quota runs 50 ms, sits parked for 50 ms, then finishes: 130 ms of wall time for 80 ms of work, a 60% latency penalty invisible to any profiler that only measures on-CPU time.

Now add monitoring. Enforcement happens per 100 ms window; your utilization graphs average over scrape intervals thousands of times longer. A service that idles most of the time but occasionally fans out hard can be throttled in a meaningful fraction of periods while its averaged utilization sits far below the limit. The average says "healthy, over-provisioned even." The p99 says otherwise. Neither is lying — they are measuring different windows.

This is exactly why Indeed's team stopped looking at utilization and defined a different health metric: throttled percentage, nr_throttled / nr_periods, read straight from the cgroup's cpu.stat file — how often the container hit the wall, regardless of how little it used on average. cpu.stat also reports throttled_time, the cumulative nanoseconds threads spent parked.

The global pool, 5 ms slices, and the bug that lived there for years

There's a second layer to the mechanism, and for years it made everything above worse than the design intended.

Quota isn't handed to CPUs directly. The kernel docs describe a two-level scheme: "A group's unassigned quota is globally tracked, being refreshed back to cfs_quota units at each period boundary. As threads consume this bandwidth it is transferred to cpu-local silos on a demand basis." The transfer granularity is kernel.sched_cfs_bandwidth_slice_us, default 5 ms: each CPU that runs one of your threads checks out a 5 ms slice from the global pool.

Consider what that means on a big node. A thread wakes on a CPU, checks out 5 ms of quota, does 1 ms of work handling its part of a request, and sleeps. That CPU's silo now holds 4 ms of your budget. Multiply by dozens of threads landing on dozens of CPUs and a large share of your quota is checked out but unspent — and in older kernels, unspent slices expired at the period boundary. The application was charged for quota it never got to use, then throttled for the remainder.

Dave Chiluk's kernel fix — "sched/fair: Fix low cpu usage with high throttling by removing expiration of cpu-local slices" — traces the underlying defect to a 2014 commit (51f2176), with a 4.18-era clock-drift fix (512ac999d275) turning it into a severe, widely-backported regression that shipped in RHEL, CentOS, and Ubuntu kernels. The failure signature was precisely the paradox this article opened with: highly threaded, non-CPU-bound applications showing high throttling rates while failing to consume their allocated quota. The fix removes slice expiration entirely — each CPU retains at most 1 ms of slack (min_cfs_rq_runtime), and the commit reports a near-30x improvement on an artificial test case (10 ms quota per 100 ms period on an 80-core machine): "This greatly improves performance of high-thread-count, non-cpu bound applications with low cfs_quota_us allocation on high-core-count machines."

For Indeed, getting the fixed kernel deployed dropped one application's worst-case response time from over 2 seconds to 30 milliseconds. Same limits, same code — the throttling had been an accounting artifact.

What to actually do about it

The kernel bug is fixed, but the structural behavior — parallel quota burn, hard parking, 100 ms accounting — is the design, not a defect. Practical consequences:

  • Alert on throttling, not utilization. nr_throttled / nr_periods and throttled_time from cpu.stat tell you what utilization graphs structurally cannot. A pod at 30% utilization with 40% of periods throttled is misconfigured, not healthy.
  • Size limits for burst width, not average draw. The limit must cover peak concurrent thread execution within a 100 ms window. Runtime thread-pool sizing (GC threads, GOMAXPROCS-style knobs) matters as much as the limit itself.
  • Know the debate about dropping limits. The long-running Kubernetes issue #67577 exists precisely because CFS quotas "can lead to unnecessary throttling, especially for well behaved tenants." Requests (which map to scheduler weight, not quota) still guarantee fair minimums without hard parking; whether to set limits at all is a genuine tradeoff between latency and noisy-neighbor containment, not settled best practice.
  • Look at CFS burst. Newer kernels add cpu.cfs_burst_us (default 0): a cgroup can bank unused quota from prior periods and spend it during spikes, with the kernel documentation describing the overrun as bounded. It's a middle path — average consumption still capped, single-window spikes forgiven.

The one-line takeaway: a Kubernetes CPU limit is not "this container may use N cores." It is "every 100 ms, this container's threads may collectively execute for N × 100 ms, and then they stop." Once you read it that way, throttled-at-low-utilization stops being a paradox and becomes the expected output of the arithmetic.

References

kubernetescfscpu-throttlingcgroupslinux-kernel