CPU Throttling or Node Saturation: What to Check Before Moving a Pod

The application is responding more slowly, and CPU throttling is rising on the graph. The first idea is to recreate the Pod on a less busy node. But idle CPUs will not help a process stopped by its own quota.

Before moving it, establish what limits execution: a CPU-time budget or contention for the available processors. Both causes can exist at once, but they require different actions.

Below are a diagnostic approach and standalone learning experiments. This is neither an analysis of a production incident nor a report of completed measurements. The scope is Linux, cgroup v2, and the standard fair scheduling class. Hardware CPU frequency reduction is not called throttling here.

Container quota is not node utilization

In Kubernetes, requests.cpu participates in node selection. On Linux, the CPU request also affects the container’s relative weight under contention. It is not an upper bound on consumption. limits.cpu sets a limit, usually implemented through a cgroup CPU quota.¹

In cgroup v2, cpu.max contains a quota and a period in microseconds. For example:

25000 100000

This is a budget of 25 ms of CPU time per 100 ms period, or 0.25 CPU. The value max instead of a number means there is no quota at this level.²

Threads in the group consume the budget, including threads running in parallel on different CPUs. Once it is exhausted, execution is throttled until the budget is replenished. The node does not have to be busy for this to happen.³

Average consumption over five minutes can be below the limit even though the quota is exhausted in individual short periods. A long averaging window hides these bursts.

Moving to another node with the same quota does not change this rule. It may change neighboring workloads or how quickly another processor completes the work, but the limit itself remains.

The opposite case also occurs. There is no finite quota, but runnable threads share busy CPUs. The application waits for CPU time without exhausting its own budget. Raising a nonexistent quota makes no sense here.

What to measure before making a change

Start with one affected container and one time interval. Averaging across the entire service can hide an overloaded replica.

First read the Pod object stored in the API, rather than only the template in Git. Then compare it with the actual cgroup configuration. Settings can be supplemented when a Pod is created, for example through LimitRange.⁴

From a Pod to its container cgroup

The following procedure covers one setup: containerd with runc, cgroup v2, and a regular container from spec.containers that does not create child cgroups. It requires SSH access to the node with sudo: commands on the node run in its PID, mount, and cgroup namespaces. Root inside a Pod does not provide the same access.

On your workstation, you need kubectl, jq, and permission to get the Pod. Set the namespace and Pod name:

NS=example
POD=example-0
kubectl -n "$NS" get pod "$POD" -o json |
  jq '{uid: .metadata.uid, node: .spec.nodeName,
       resources: [.spec.containers[] | {name, resources}],
       containers: [.status.containerStatuses[]? |
         {name, containerID, restartCount, state}]}'

Select the container by name. Record the Pod UID, node, full containerID, restartCount, and state.running.startedAt. Continue only for a running container with an ID in the form containerd://….⁵

Connect to that node over SSH and open Bash with sudo -i. The node needs jq, crictl, findmnt and taskset from util-linux, and mpstat from sysstat to report CPU utilization. In this setup, the full cgroup v2 hierarchy is mounted at /sys/fs/cgroup, and the cpu and cpuset controllers and PSI are available. Configure crictl for the actual containerd endpoint in /etc/crictl.yaml beforehand.⁶

set -euo pipefail
CID='<full ID without containerd://>'
findmnt -n -o FSTYPE,FSROOT,TARGET -T /sys/fs/cgroup
test "$(stat -fc %T /sys/fs/cgroup)" = cgroup2fs
test "$(findmnt -n -o FSROOT -T /sys/fs/cgroup)" = /
pid="$(crictl inspect -o json "$CID" |
  jq -er 'select(.status.state == "CONTAINER_RUNNING")
          | .info.pid | numbers | select(. > 0)')"
group="$(awk -F: '$1 == "0" {print $3}' "/proc/$pid/cgroup")"
case "$group" in
  /*) test "$group" != / ;;
  *) echo "Container cgroup is unavailable" >&2; exit 1 ;;
esac
cg="/sys/fs/cgroup$group"
test -r "$cg/cpu.stat"
printf 'PID=%s cgroup=%s\n' "$pid" "$cg"

The expected findmnt values are cgroup2, /, and /sys/fs/cgroup. The .info.pid field is a containerd extension, not a general CRI contract.⁷ The path comes from /proc for the discovered PID; no systemd or cgroupfs directory layout is assumed. In another mount or cgroup namespace, check hierarchy visibility before reusing this procedure.⁸

Check the entire accessible hierarchy

A container’s cpu.max = max does not mean there are no limits above it. A parent cgroup can exhaust its shared budget and stop a child even if the child still has quota remaining.³

In the same root session, collect data for the container and all its ancestors, including the Pod cgroup. Each level needs a pair of snapshots over a comparable interval:

snapshot_hierarchy() (
  set -eu
  dir="$cg"
  while :; do
    test -d "$dir"
    printf '\n%s uptime_s=' "$dir"
    cut -d ' ' -f 1 /proc/uptime
    for file in cpu.max cpu.stat cpu.pressure cpuset.cpus.effective; do
      printf '%s: ' "$file"
      if test -r "$dir/$file"; then
        cat "$dir/$file"
      else
        printf 'unavailable\n'
      fi
    done
    test "$dir" = /sys/fs/cgroup && break
    dir="$(dirname "$dir")"
  done
)
snapshot_hierarchy
sleep 5
snapshot_hierarchy

For each path, compare its cpu.max and the changes in usage_usec, nr_periods, nr_throttled, and throttled_usec from cpu.stat. A finite quota shows the configuration; growing throttling counters show that it took effect. These counters describe the level’s own quota, so no growth at the container level does not rule out ancestor throttling. An ancestor’s usage_usec includes all its descendants, not just the selected container.²

The uptime_s value is in seconds since the node booted. Calculate the interval from the timestamps for the same path: reading files adds time to sleep 5, and the hierarchy walk is not atomic. The actual hierarchy root normally has no cpu.max. Do not treat other unavailable files or fields as zeros. If some ancestors are hidden or inaccessible, these data cannot rule out throttling by their quotas.²⁸

After measuring, repeat the Pod query on the workstation and crictl inspect on the node. Check that the UID, container ID, start time, restart count, PID, and path in /proc/$pid/cgroup are unchanged. If the container was replaced, the cgroup disappeared, or counters reset, start a new measurement.

Also establish which CPUs the process can use. cpuset.cpus.effective shows the cgroup’s effective set, and thread affinity can narrow it further. An idle core outside that set will not help the thread.²⁹

In the same node session, read the affinity of every thread in each container process:

cat "$cg/cpuset.cpus.effective"
while read -r process; do
  taskset -apc "$process"
done < "$cg/cgroup.procs"

In another terminal on that node, run mpstat -P ALL 1 5 alongside the snapshots. Match the allowed CPU numbers to the individual CPU rows, rather than the all average. taskset -apc reads affinity without changing it. %iowait does not mean the CPU is busy computing; on a VM, %steal reports time taken by the hypervisor.¹⁰

Distinguish consumption from waiting

CPU utilization shows work performed. PSI helps assess waiting: /proc/pressure/cpu on the node and cpu.pressure in the relevant cgroup. The some field reflects the fraction of time when at least some tasks were delayed by resource unavailability. For a short experiment, compare the increase in total, rather than only the smoothed avg10.¹¹

Cgroup CPU pressure can also rise under quota throttling: high PSI alone does not prove contention with neighboring workloads.¹² Nor does it prove the cause of latency for a particular request. Combine the runnable thread queue, utilization of the CPUs available to those threads, cgroup limits, and application behavior.

If one allowed CPU is busy while the others are idle, the entire node is not necessarily saturated. Do not select a node by average utilization without checking the CPU set available to the application.

What the percentage of throttled periods means

cpu.stat contains the cumulative counters nr_periods, nr_throttled, and throttled_usec. Diagnosis requires their changes over an interval, rather than their totals since the cgroup was created.²

cAdvisor can export the corresponding metrics to Prometheus. Here is an example query for one illustrative container:¹³

100 *
rate(container_cpu_cfs_throttled_periods_total{
  namespace="example", pod="example-0", container="app"
}[5m])
/
rate(container_cpu_cfs_periods_total{
  namespace="example", pod="example-0", container="app"
}[5m])

Add cluster and collection-source labels to the selector so that it does not mix different containers or duplicate series. rate() handles counter resets, but it cannot correct an incorrect series selection.¹⁴

The result shows the fraction of counted periods in which throttling occurred. It does not show the fraction of lost requests or the percentage of CPU capacity the application was denied. If throttling occurred briefly in every period, the value can still approach 100%.¹³

container_cpu_cfs_throttled_seconds_total also cannot automatically be converted into the number of cores the application needs: it measures throttling time, rather than computational demand. The metric describes the mechanism, rather than its impact on users.

If the period counter has not increased, the fraction is undefined. An empty query result is not zero either. First check metric availability, labels, and collection. And a lack of growth in a child cgroup’s counters does not remove the need to check its ancestors.

Calculating changes from two snapshots

These are illustrative numbers, not a completed measurement. The same cgroup has one available CPU and an unchanged cpu.max = 25000 100000. The snapshots are exactly 5 s, or 5 000 000 µs, apart, with no container restart:

Average consumption is 1 000 000 / 5 000 000 = 0.2 CPU: the group used one second of CPU time in five seconds. The fraction of throttled periods is 20 / 50 × 100 = 40%. The increase in some total gives 1 500 000 / 5 000 000 × 100 = 30% of the interval with CPU pressure; it is not the sum of every thread’s waiting time.¹¹

The group’s own quota did take effect, even though average consumption was below its 0.25 CPU budget. These data do not show the fraction of lost requests, the required number of CPUs, or the quota’s effect on application latency. They also do not rule out an ancestor limit or prove contention with neighboring workloads. A decision needs ancestor snapshots, utilization of the allowed CPUs, and application metrics for the same interval.

Two experiments that separate the causes

A separate Linux VM without production workloads is enough to isolate the mechanisms. It needs at least two available logical CPUs, systemd with AllowedCPUs support, cgroup v2 with the cpu and cpuset controllers, enabled PSI, and an installed stress-ng. The lab does not need Kubernetes: the experiments test kernel behavior, rather than the Pod scheduler.

Before starting, verify that parent cgroups have no limiting quota. Record the kernel, systemd, and stress-ng versions. On a VM, account for hypervisor contention as well: unstable vCPU availability makes the lab unsuitable for comparison.

Choose one allowed CPU. The example uses CPU 1; it must be online and available to the test services. This confines the load, but does not reserve that processor exclusively for the experiment.¹⁵

Prepare this function in the first terminal:

CPU=1
STRESS_NG="$(command -v stress-ng)"
test -n "$STRESS_NG" || { echo "stress-ng is not installed" >&2; exit 1; }
test "$(stat -fc %T /sys/fs/cgroup)" = cgroup2fs || exit 1

run_cpu_case() {
  sudo systemd-run --unit="$1" --wait --pipe --collect \
    --property=Type=exec \
    --property="AllowedCPUs=$CPU" \
    --property=CPUWeight=100 \
    --property="CPUQuota=$3" \
    --property=CPUQuotaPeriodSec=100ms \
    --property=RuntimeMaxSec=30s \
    --property=TimeoutStopSec=3s \
    "$STRESS_NG" --cpu "$2" --cpu-method int64 \
      --timeout 20s --metrics-brief
}

CPUQuota=25% limits the service to a quarter of one CPU, rather than a quarter of the entire VM. An empty value removes the quota only from the service itself.¹⁵ stress-ng runs the specified number of CPU workers for 20 seconds. The service runtime limit also stops an experiment if it hangs.¹⁶¹⁷

Before the first run, prepare the measurement function in the second terminal. It takes two readings with a five-second pause:

measure_cpu_case() (
  set -eu
  unit="$1"
  group="$(systemctl show "$unit" --property=ControlGroup --value)"
  case "$group" in
    /*) test "$group" != / ;;
    *) echo "Service cgroup is unavailable" >&2; exit 1 ;;
  esac
  cg="/sys/fs/cgroup$group"

  snapshot() {
    date -Ins
    for file in cpu.max cpu.stat cpu.pressure cpuset.cpus.effective; do
      printf '\n%s\n' "$file"
      cat "$cg/$file"
    done
  }

  snapshot
  sleep 5
  snapshot
)

The service’s cgroup may disappear after it finishes. A read error in that case means a missed measurement, rather than zero counters. Repeat the experiment and take readings while it is running.

Use the difference between snapshot timestamps for calculations. run_cpu_case waits for the service to finish, so measure in the second terminal while the first is running the load.

Start with a baseline run

One worker without its own quota. In the first terminal:

run_cpu_case cpu-lab-baseline 1 ""

In the second terminal, immediately after starting the run:

measure_cpu_case cpu-lab-baseline.service

Experiment 1: a quota on an idle CPU

After the baseline run finishes, execute in the first terminal:

run_cpu_case cpu-lab-quota 1 "25%"

In the second terminal, immediately after starting the run:

measure_cpu_case cpu-lab-quota.service

The expected difference from the baseline is a finite quota and growing throttling counters. One continuously running process wants more CPU time than its group is allowed. A second load-generating process is not needed for this limit to take effect.

To check the hypothesis, repeat the run with an empty quota, the same number of workers, and the same CPU. Only one factor changes: the execution-time budget.

Experiment 2: contention without a service quota

After the previous experiment finishes, execute in the first terminal:

run_cpu_case cpu-lab-contention 2 ""

In the second terminal, immediately after starting the run:

measure_cpu_case cpu-lab-contention.service

Two CPU workers now share one allowed CPU. With no ancestor quotas, CPU waiting is expected to grow without an increase in quota-throttling counters. Compare this with the baseline run, where one worker used the same CPU.

This experiment models contention within the available CPU set, rather than load on every core in the node. It also shows why idle CPUs on a node-wide graph do not prove that a particular process has resources available.

Wait for the service to finish after each run. To stop a run early, use sudo systemctl stop with the exact name of the current test service.

How to read the result

These are expected signs, rather than measured results. If they do not appear, check the effective cpuset, ancestor quotas, background load, and scheduling settings. Do not force the conclusion to match the table.

The stress-ng operation counters help compare identical tests. They do not replace application latency and throughput measurements. The experiments explain the mechanism, but do not prove that it caused a particular degradation.

Why Kubernetes will not move this Pod on its own

A Pod object is assigned to a node once. “Moving” usually means terminating the old Pod and creating a new one with a different UID. Restarting a container alone does not change the node.¹⁸

The standard scheduler considers resource requests and placement constraints. It is not a continuous load balancer for running Pods based on CPU utilization graphs.¹ The standard node-pressure eviction signals include memory, disk resources, and PIDs, but not CPU pressure.¹⁹

Do not expect automatic relocation solely because CPU throttling rises. Additional automation is possible, but it will still need to distinguish the causes of delay.

Choose an action based on the confirmed cause

If the application’s own threads continue sharing the same narrow cpuset or affinity, moving to another node will not remove that constraint. If CPU observations are sufficient but neither quotas nor contention explain the degradation, investigate application waits, such as locks or I/O. These observations do not yet justify changing the CPU limit or placement.

Raising a limit without assessing the node is risky: it can turn throttling in one container into greater contention for its neighbors. Additional replicas on the same busy CPUs do not create more computing resources either.

Before recreating a Pod, check whether interruption is acceptable, its owning controller, storage constraints, and the availability of a suitable target node. Planned eviction uses an agreed process that accounts for PodDisruptionBudget; ordinary Pod deletion does not provide equivalent protection.²⁰ The new Pod can also land on the same node if placement rules allow it.

Assess the effect under comparable incoming load. Change one factor at a time: limit, placement, or replica count. Compare latency, errors, and completed requests after warm-up. Eliminating throttling without improving application behavior does not prove the problem is solved.

Takeaway

Before moving a Pod, answer two separate questions: is it exhausting its available CPU quota, and is it waiting for busy processors? Neither average node utilization nor the percentage of throttled periods explains this on its own.

Moving helps when it changes a confirmed constraint. If the same CPU quota remains the constraint, address that quota before changing the Pod’s location.