Low average CPU, slow requests: diagnosing cgroup v2 cpu.max throttling

A container has a one-core CPU limit, the dashboard shows average usage well below that, and yet request latency occasionally jumps. In that situation, CPU bandwidth control throttling in cgroups is worth checking.

This post follows the Linux kernel documentation, version 7.3.0: Control Group v2 and CFS Bandwidth Control. It covers what cpu.max limits, why a group can stall while its average stays low, and how to confirm it by reading cpu.stat. The diagnosis part only reads files and changes no settings.

cpu.max limits CPU time per period

In cgroup v2, cpu.max is a read-write file on non-root cgroups, in this format:

$MAX $PERIOD

The group may consume up to $MAX in each $PERIOD, both in microseconds. The default is max 100000, and max in the $MAX position means no limit. Writing a single number updates only $MAX.

The important part is that $MAX is not wall-clock time. It is CPU time summed over every thread in the group on every CPU. The sched-bwc examples describe it this way: a 1000ms quota over a 500ms period gives “2 CPUs worth” of runtime, and a 10ms quota over a 50ms period is 20% of one CPU. So cpu.max of 100000 100000 is what people usually call a one-core limit.

According to the sched-bwc document, quota is handed to per-CPU run queues in slices as the group’s threads become runnable. Once all quota has been assigned, threads asking for more are throttled, and throttled threads cannot run again until the next period, when the quota is replenished.

Why a low average can still stall

A calculation follows directly from that definition. Take a group with cpu.max set to 50000 100000 (0.5 CPU) and eight threads working at the same time on eight CPUs. Idealized, the 50ms quota is used up in about 6.25ms of wall time, and the group’s threads then wait for the remaining 93.75ms or so of the period.

The group’s average usage is still at or below 0.5 CPU. But a request that arrives during the wait is not served until the next period, and a dashboard that averages per second will not show stalls inside a period. These numbers just apply the definition; in practice the slice distribution described below can shift the timing somewhat.

The limit applies per period, not on average, and the more threads run at once, the faster the quota runs out in wall-clock terms. If the gap between CPU time and wall time is unfamiliar, the real, user, and sys time post may help.

Two ways a group gets throttled

The sched-bwc document lists two cases:

1. the group fully consumes its own quota within a period
2. a parent’s quota is fully consumed within its period

In the second case, a child that still has runtime left cannot use it until the parent’s runtime is refreshed. Over-subscription is also explicitly allowed in a hierarchy: the children’s limits may add up to more than the parent’s. Looking only at one child’s cpu.max and concluding there is headroom can therefore miss the cause.

Fields to read in cpu.stat

cpu.stat is a read-only flat-keyed file. It exists whether or not the cpu controller is enabled, but the fields it contains differ.

Field	When present	What it counts
usage_usec, user_usec, system_usec	always	CPU time of all processes, including descendant cgroups
nr_periods	controller enabled	enforcement intervals that have elapsed
nr_throttled	controller enabled	times the group has been throttled
throttled_usec	controller enabled	total time the group’s entities were throttled
nr_bursts, burst_usec	controller enabled	periods in which a burst occurred, and cumulative time used above quota

A few points to keep in mind when reading it:

• The five bandwidth fields count only processes under the fair-class scheduler. The first three, usage_usec and the rest, count all processes.
• The five bandwidth fields are not hierarchical. They count only throttling caused by this cgroup’s own limit, not throttling inherited from ancestor cgroups.
• To include throttling inherited from an ancestor, read throttled_usec in cpu.stat.local. The 7.3.0 documentation describes it as the throttling time actually incurred by the cgroup’s own run queues, which may include throttling inherited from ancestor limits. This post calls it local runqueue throttling time. A rising value shows that this cgroup’s local run queues were throttled, but it is time accumulated by run queues, not a direct measure of request latency or the length of one wall-clock outage. Some kernels may not have this file, so check that it exists first.
• In cgroup v1, the sched-bwc statistics report throttled_time in nanoseconds, while the cgroup v2 documentation states that all time durations in the cpu interface files are in microseconds. Comparing numbers across the two versions without converting puts you off by a factor of 1000.

nr_throttled is cumulative and hard to judge on its own. Reading it twice at a fixed interval and looking at the difference works better. The sched-bwc document says that for CPU-bound work held at its limit, nr_periods is expected to roughly equal nr_throttled. For a service that is mostly idle with occasional spikes, the gap between the two can be small; what matters there is whether throttled_usec keeps rising.

A read-only diagnostic script

The script below only reads cgroup files. It finds the cgroup v2 path of a given PID from the 0:: line of /proc/<PID>/cgroup, prints cpu.max for that cgroup and its ancestors up to and including the mount root, then computes how cpu.stat changed over an interval. It stops with an error instead of guessing when the interval is not a positive integer, /proc/<PID>/cgroup cannot be read, there is not exactly one absolute 0:: path, CG_ROOT or the cgroup path fails the boundary checks described below, the resulting cpu.stat cannot be read, the cgroup directory disappeared or was replaced during the interval, a sampled value is missing or not a number, or a required counter went backward between the two samples. A counter going backward means the cgroup was reset or recreated, so that interval cannot be used.

To detect a replaced directory, the script compares the inode number that ls -di reports for the cgroup directory before the first sample and after the second. The -d and -i options are standard ls options, but treating the inode number as the identity of a cgroup directory is a Linux cgroupfs assumption: it relies on a recreated directory getting a different inode number, which the kernel documentation cited here does not cover. That is why the script also refuses counters that went backward.

It assumes the cgroup2 filesystem is mounted at /sys/fs/cgroup. That can vary between systems, so check with findmnt -t cgroup2 first and set CG_ROOT to an absolute path if it differs.

The script does not trust either path as given. It resolves CG_ROOT with cd -P and pwd -P, which only read the filesystem, and stops unless the result is an existing directory other than / that contains cgroup.controllers, a file the cgroup v2 documentation says exists in every cgroup. It also rejects a 0:: path with . or .. components, resolves the cgroup directory the same way, and stops if it does not exist or is not CG_ROOT itself or a directory below it. It never reads cgroup files outside CG_ROOT.

Inside a container with its own cgroup namespace, /proc/<PID>/cgroup shows paths relative to the cgroupns root, so the PID’s own cgroup can appear as 0::/. A cgroup2 mount made inside that namespace has the cgroupns root as its filesystem root. The cgroupns root is the cgroup the namespace was created in, so it is usually not the host’s root cgroup and can have its own cpu.max. That is why the script also prints cpu.max at CG_ROOT itself when it is readable, but never looks above it. Limits on host cgroups above the namespace root are not visible from inside.

#!/bin/sh
# usage: sh cpu-throttle.sh [PID] [SECONDS]
pid=${1:-self}
secs=${2:-10}
root=${CG_ROOT:-/sys/fs/cgroup}

die() { echo "error: $*" >&2; exit 1; }
isnum() { case $1 in ''|*[!0-9]*) return 1 ;; esac; return 0; }
field() { awk -v k="$2" '$1 == k { print $2 }' "$1" 2>/dev/null; }
dirid() { ls -di "$1" 2>/dev/null | awk 'NR == 1 { print $1 }'; }
realdir() { (cd -P -- "$1" 2>/dev/null && pwd -P); }

# Positive integer without a leading zero (shell arithmetic reads 010 as octal).
case $secs in
  ''|0*|*[!0-9]*) die "SECONDS must be a positive integer: $secs" ;;
esac

# CG_ROOT: absolute, resolved to an existing cgroup v2 directory other than /.
case $root in
  /*) ;;
  *) die "CG_ROOT must be an absolute path: $root" ;;
esac
r=$(realdir "$root") || die "CG_ROOT is not an existing directory: $root"
[ "$r" != / ] || die "CG_ROOT must not be /"
[ -e "$r/cgroup.controllers" ] || die "CG_ROOT is not a cgroup v2 directory: $root"
root=$r

# Require exactly one non-empty absolute cgroup v2 path, without . or .. components.
f=/proc/$pid/cgroup
[ -r "$f" ] || die "cannot read $f"
rel=$(awk 'index($0, "0::") == 1 { n++; p = substr($0, 4) } END { if (n == 1) print p }' "$f")
case $rel in
  /*) ;;
  *) die "no single cgroup v2 path (0::/...) in $f" ;;
esac
case $rel/ in
  */./*|*/../*) die "cgroup path contains . or ..: $rel" ;;
esac
cg=$(realdir "$root$rel") || die "cgroup directory does not exist: $root$rel"
# The resolved cgroup must be CG_ROOT itself or below it.
case $cg in
  "$root"|"$root"/*) ;;
  *) die "cgroup $cg is outside CG_ROOT $root" ;;
esac
[ -r "$cg/cpu.stat" ] || die "cannot read $cg/cpu.stat"
echo "cgroup: $cg"

# Directory identity (inode number), compared again after the second sample.
id1=$(dirid "$cg")
isnum "$id1" || die "cannot read the inode number of $cg"

# Limits on this cgroup and its ancestors, up to and including CG_ROOT, never above it.
# The host root cgroup has no cpu.max; a cgroup namespace root mounted at CG_ROOT can have one.
d=$cg
while :; do
  [ -r "$d/cpu.max" ] && echo "  $(cat "$d/cpu.max")  burst=$(cat "$d/cpu.max.burst" 2>/dev/null)  <- $d"
  if [ "$d" = "$root" ] || [ "$d" = "/" ]; then break; fi
  d=$(dirname "$d")
done

p1=$(field "$cg/cpu.stat" nr_periods)
if [ -z "$p1" ]; then
  if [ "$cg" = "$root" ] && [ ! -e "$cg/cpu.max" ]; then
    echo "no nr_periods: CG_ROOT has no cpu.max (the host root cgroup, or a namespace root without the cpu controller), so there is no bandwidth limit to diagnose here."
  else
    echo "no nr_periods: the cpu controller is not enabled for this cgroup."
  fi
  exit 0
fi
t1=$(field "$cg/cpu.stat" nr_throttled)
u1=$(field "$cg/cpu.stat" throttled_usec)
c1=$(field "$cg/cpu.stat" usage_usec)
l1=$(field "$cg/cpu.stat.local" throttled_usec)

sleep "$secs"

p2=$(field "$cg/cpu.stat" nr_periods)
t2=$(field "$cg/cpu.stat" nr_throttled)
u2=$(field "$cg/cpu.stat" throttled_usec)
c2=$(field "$cg/cpu.stat" usage_usec)
l2=$(field "$cg/cpu.stat.local" throttled_usec)

id2=$(dirid "$cg")
[ "$id2" = "$id1" ] || die "$cg disappeared or was replaced during the interval (inode $id1 -> ${id2:-none})"

for v in "$p1" "$p2" "$t1" "$t2" "$u1" "$u2" "$c1" "$c2"; do
  isnum "$v" || die "missing or non-numeric value in $cg/cpu.stat (did the cgroup go away?)"
done

# A counter that went backward means a reset or recreation; the interval is invalid.
if ! { [ "$p2" -ge "$p1" ] && [ "$t2" -ge "$t1" ] && [ "$u2" -ge "$u1" ] && [ "$c2" -ge "$c1" ]; }; then
  die "a cpu.stat counter went backward (cgroup reset or recreated?); interval discarded"
fi

echo "over ${secs}s:"
echo "  throttled in $((t2 - t1)) of $((p2 - p1)) periods"
echo "  throttled time $(((u2 - u1) / 1000)) ms (this cgroup's own limit)"
if isnum "$l1" && isnum "$l2" && [ "$l2" -ge "$l1" ]; then
  echo "  local runqueue throttling time $(((l2 - l1) / 1000)) ms (cpu.stat.local, may include ancestors, not wall-clock stall time)"
elif isnum "$l1" && isnum "$l2"; then
  echo "  local runqueue throttling time: unavailable (cpu.stat.local throttled_usec went backward, reset or recreated)"
else
  echo "  local runqueue throttling time: unavailable (no numeric cpu.stat.local throttled_usec in both samples)"
fi
echo "  CPU used $(((c2 - c1) / 1000)) ms = average $(((c2 - c1) / secs / 10000))% CPU"

The percentage on the last line treats one CPU as 100%, and integer division drops the fraction. Read the output like this:

• If average usage is well below the limit but the throttle count keeps climbing, this cgroup’s own limit is throttling it in some periods even though the interval average looks low. The counters carry no timestamps inside a period, so they cannot tell whether the quota ran out early in each period or late in some periods followed by idle time. Treat early-period exhaustion as a hypothesis to check with finer-grained tracing or by lining the throttling up with request and workload timing.
• If this cgroup’s own throttled_usec barely moves while the local runqueue throttling time from cpu.stat.local rises, suspect an ancestor’s limit. Compare against the ancestor cpu.max list printed at the top. That value says the local run queues were throttled; it is not request latency.
• If there is no nr_periods line, there are no bandwidth statistics. For a non-root cgroup, that means the cpu controller is not enabled for it. If the PID is at CG_ROOT and there is no cpu.max there, the script says so instead: either it is the host root cgroup, which has no cpu.max, or it is a namespace root without the cpu controller, and in both cases there is no bandwidth limit on it for this diagnostic to measure.

Time spent waiting for CPU in general also shows up in cpu.pressure in the same directory, which accounts for all processes in the cgroup. How to read it is covered in the PSI post.

Options for fixing it, and what they cost

Unlike the diagnosis, changing settings affects running work immediately. If a tool created the cgroup (a container runtime, a service manager), changing the setting through that tool is the safer route. The options below follow from the behavior the kernel documentation describes.

Raise the quota. This is the simplest. As the calculation above shows, though, with many threads running at once, a quota sized to average usage can still leave stalls within a period.

Change the period. The sched-bwc examples explain that a longer period allows more burst capacity, while a shorter one gives up burst capacity in exchange for more consistent latency. At the same ratio, 50000 100000 and 5000 10000 behave differently. The same document’s v1 interface section gives 1ms as the minimum for quota and period, and 1s as the maximum period.

Use cpu.max.burst. The default is 0 and the range is 0 to $MAX. The sched-bwc document describes burst as letting a group accumulate unused bandwidth up to the configured value and then run above its quota; at 0 nothing accumulates and the traditional behavior is unchanged. It also states the cost: more interference with other users of the system. Whether bursts actually happened shows up in nr_bursts and burst_usec in cpu.stat.

Use a weight instead of a cap. cpu.weight (default 100, range 1 to 10000) splits CPU among active sibling cgroups in proportion to their weights. Only children that can use CPU at the moment take part, so it is work-conserving: a group can take the share its siblings are not using. If what you need is a ratio under contention rather than a hard ceiling, this may fit better.

Slices and machines with many cores

According to sched-bwc, the kernel moves quota from the global pool to per-CPU “silos” one slice at a time. The slice size is /proc/sys/kernel/sched_cfs_bandwidth_slice_us, 5ms by default. A larger value reduces transfer overhead; a smaller one allows finer-grained consumption.

A slice assigned to a CPU does not expire. If every thread on that CPU becomes unrunnable, all but 1ms of it may be returned to the global pool; the 1ms comes from min_cfs_rq_runtime, a compile-time value. As a result, a highly threaded workload that is not CPU bound can briefly run past its quota by the unused slice left on each CPU, typically at most 1ms per CPU. The document notes that average usage is still strictly limited to the quota, just over a longer window than a single period. Keep this in mind when cpu.stat numbers do not line up exactly with the definition.

Scope and limits

• The cgroup v2 documentation warns that the cpu controller does not yet support bandwidth control of realtime processes. The cpu.max behavior in this post applies to processes under the fair-class scheduler (and BPF schedulers that implement the corresponding callback). SCHED_FIFO, SCHED_RR, and SCHED_DEADLINE processes are handled differently.
• The documentation says CPU distribution is defined only in terms of time and does not account for the frequency the CPU runs at. How much work fits into the same quota can vary with frequency.
• This post is based on kernel documentation version 7.3.0, checked on 2026-10-22. Files such as cpu.stat.local may be missing on older kernels. Check the documentation for the kernel you run alongside the files actually present.
• The 0.5 CPU, eight-thread example in the text is a calculation from the definition, not a measurement.

Sources

• Linux kernel documentation 7.3.0, Control Group v2 (checked 2026-10-22)
• Linux kernel documentation 7.3.0, CFS Bandwidth Control (checked 2026-10-22)