A Ready Node Can Still Drift from Its Baseline

A green Ready status and a correct node configuration are not the same guarantee. I ran into that distinction while standardizing kubelet settings across several node classes in a Kubernetes cluster I operated.

Each class had a configuration baseline stored in Git. It defined resource reservations, eviction thresholds, and image garbage collection settings. Kubernetes reported the nodes as Ready, yet a separate check found that the local /var/lib/kubelet/config.yaml on some of them no longer matched the expected file.

Nothing was visibly broken at that moment. The risk was that nodes which looked identical to the control plane could react differently once memory or disk became scarce.

Two different contracts

The Ready=True condition means that Kubernetes considers a node healthy and ready to accept Pods.¹ This is a health contract between the node and Kubernetes. It is not a compliance contract between the host and an organization-specific baseline.

The control plane has no built-in knowledge of my node classes, the Git revision expected for each class, or the checksum expected on a particular host. The following state is therefore entirely valid:

Ready=True
desired_config != actual_config

The kubelet can start with a valid configuration, keep publishing node status, and continue running workloads even when that configuration is not the one the platform team intended to deploy.

Drift appears at the edges

Kubelet configuration often controls what happens near resource limits rather than during normal operation. Two nodes can behave the same for weeks and diverge only when a threshold is crossed.

Different kubeReserved or systemReserved values change the calculation of Node Allocatable. With enforcement configured, the kubelet can also account for those reservations in the relevant system cgroups.² Different eviction settings change when the kubelet reports pressure, reclaims node-level resources, and begins terminating Pods.³ Image garbage collection thresholds can create a similar difference in disk behavior.

This makes configuration drift easy to miss in routine checks. It is not necessarily an active outage. It is a hidden difference in the rules that will govern the next stressful condition.

Making the baseline observable

I turned the Git baseline into an explicit check by deploying an internal agent as a DaemonSet. The DaemonSet kept a Pod on every selected node, but the agent was not a controller: it had no continuous reconciliation loop, polling, or automatic remediation. Its mode was defined in the DaemonSet configuration, and another verification was triggered by a controlled update of its Pods. The agent identified the node class from a label, selected the corresponding desired file, read the actual host configuration, and compared their checksums. When they differed, it printed a unified diff so that the changed settings were visible.

A shortened result looked like this:

node: <node-name>
class: worker-large
desired_sha256: <desired-checksum>
actual_sha256: <actual-checksum>
status: drift_detected

Detection did not trigger automatic remediation. I kept check, apply, and rollback as explicit operations. A change was first tested on one node, followed by a kubelet health check, before the rollout was expanded.

That choice was deliberate. Updating kubelet configuration changes a host file and restarts a system service. For an infrequent and high-impact operation, a controlled rollout was a better fit than a background controller continuously enforcing the baseline.

Verify the real configuration source

A checksum is only meaningful when it covers the configuration the process actually consumes. The kubelet configuration path is selected with --config, and command-line flags override matching values from the file.⁴

The configuration sources in this cluster were known and constrained, so checking /var/lib/kubelet/config.yaml was sufficient. In another environment, the kubelet startup arguments must be inspected first. Otherwise, a matching file can provide the same false confidence as a green Ready status.

A more useful node health model

I now treat node verification as three separate questions:

Each question catches a different class of failure. A checksum comparison can prove that the deployed file matches Git, but it cannot prove that the baseline itself is correct. Likewise, Ready can prove current operability without saying anything about configuration consistency.

Conclusion

Ready remains an important signal, but its scope is narrower than “the node is fully correct.” It tells me that Kubernetes currently considers the node healthy enough to run workloads. It does not tell me that the kubelet matches the platform baseline or that peer nodes will make the same decisions under pressure.

When consistent node behavior is a requirement, configuration compliance needs its own observable signal. A node can be Ready while already carrying a different set of operational rules from the rest of its class.