A Ready Node Can Still Drift from Its Baseline

A green Ready status and a correct node configuration are not the same guarantee. Consider several node classes with configuration baselines in Git. The baselines define resource reservations, eviction thresholds, and image garbage collection settings. Actual host files can differ from those baselines while Kubernetes continues reporting the nodes as ready.

This scenario lets us examine the problem without describing a particular cluster or operations tool. A node can work now yet react differently from its peers once memory or disk becomes scarce.

Two different contracts

The Ready=True condition means that Kubernetes considers a node healthy and ready to accept Pods.¹ This is a health contract between the node and Kubernetes. It is not a compliance contract between the host and an organization-specific baseline.

The control plane has no built-in knowledge of the file chosen as the baseline, the Git revision expected for a class, or the checksum expected on a particular host. The following state is therefore entirely valid:

Ready=True
desired_config != actual_config

The kubelet can start with a valid configuration, keep publishing node status, and continue running workloads even when that configuration is not the selected baseline.

Drift appears at the edges

Kubelet configuration often controls what happens near resource limits rather than during normal operation. Two nodes can behave the same for weeks and diverge only when a threshold is crossed.

Different kubeReserved or systemReserved values change the calculation of Node Allocatable. With enforcement configured, the kubelet can also account for those reservations in the relevant system cgroups.² Different eviction settings change when the kubelet reports pressure, reclaims node-level resources, and begins terminating Pods.³ Image garbage collection thresholds can create a similar difference in disk behavior.

This makes configuration drift easy to miss in routine checks. It is not necessarily an active outage. It is a hidden difference in the rules that will govern the next stressful condition.

Making the baseline observable

Detecting drift requires determining the expected configuration independently, reading the file actually in use, and comparing the two. The result should expose not only a mismatch but also enough information about the differences to support the next decision.

At the behavior level, the contract is:

expected configuration is established in advance
→ actual configuration is read without changing it
→ match, mismatch, and read failure are reported separately

A read failure must not become “no drift.” Otherwise, the check would confirm a state it never observed.

Detection is useful to separate from remediation. Comparing files should not itself restart kubelet or change host configuration. Applying a configuration, verifying service recovery, and returning to a previous state require a separate controlled process.

How the check reaches each node is an implementation choice. A particular agent and its interface are not needed to explain the boundary between health and baseline compliance.

Verify the real configuration source

A checksum is only meaningful when it covers the configuration the process actually consumes. The kubelet configuration path is selected with --config, and command-line flags override matching values from the file.⁴

Inspect the startup arguments and identify configuration sources before comparing files. A match at a familiar path does not prove that the process uses those settings. Otherwise, a matching file can provide the same false confidence as a green Ready status.

A more useful node health model

It helps to treat node verification as three separate questions:

Each question catches a different class of failure. A checksum comparison can prove that the deployed file matches Git, but it cannot prove that the baseline itself is correct. Likewise, Ready can prove current operability without saying anything about configuration consistency.

Conclusion

Ready remains an important signal, but its scope is narrower than “the node is fully correct.” It indicates that Kubernetes currently considers the node healthy enough to run workloads. It does not establish that the kubelet matches the selected baseline or that peer nodes will make the same decisions under pressure.

When consistent node behavior is a requirement, configuration compliance needs its own observable signal. A node can be Ready while already carrying a different set of operational rules from the rest of its class.