A simplified kubelet configuration change looks straightforward: save a backup, write the new file, restart the service. If something goes wrong, restore the backup and restart again.
The problem is the phrase “goes wrong.” Did writing the file fail? Did kubelet fail to start? Did the node fail to return to service? Or did recovery succeed while the automation failed to record the final status?
The more useful question is: what state did the node end up in, and what evidence proves it?
A rollback command alone does not answer that. Rollback needs its own contract: trigger conditions, recovery point, success criteria, and behavior after partial execution.
A backup is not yet a known-good state
It helps to distinguish four states:
- Desired configuration — what we intend to apply
- Configuration on disk — what is currently stored in the file
- Active configuration — the parameters used by the running kubelet
- Last-known-good configuration — a state that already passed the agreed checks
These may match during normal operation. During a change, they do not have to.
After replacing the file but before restarting kubelet, the new configuration is already on disk while the process is still running with the old one. Effective kubelet settings also depend on startup arguments: command-line flags other than feature gates take precedence over corresponding values in the configuration file. Feature gates follow a separate merge order.1
So the contents of one YAML file cannot automatically be treated as the active configuration.
This leads to the first rollback requirement: we need to know not only which file was saved, but also why that file is a valid recovery point.
If a later run backs up an already broken or still-unverified file, a newer timestamp does not make that copy better than the previous one.
And last-known-good is not a permanent guarantee. It is a configuration verified in a particular environment. If the kubelet version, startup arguments, or other node state changes at the same time, previous verification may no longer be sufficient.
Rollback preparation starts before the change
The most uncomfortable failure point is between modifying the system and recording information about that modification.
The file may already be replaced while the automation has not yet persisted where to recover from. Or kubelet may already be restarted while execution stops before verification.
Before the first mutating action, the recovery information therefore needs to exist:
- which configuration was the starting point
- where its backup is stored
- how to verify the backup’s integrity
- which change operation it belongs to
The backup itself also needs verification before use. A path is not enough: the file may be missing, unreadable, or contain unexpected data.
The target configuration needs separate preflight checks. Valid YAML proves syntax, not compatibility with the kubelet version or the node environment.
Atomic file replacement is useful, but it solves only part of the problem. rename can atomically replace one file within a filesystem, but it does not turn file replacement, service restart, and operation-state persistence into one transaction.2
Reliability here comes from recognizing intermediate states and recovering from them, not from pretending they do not exist.
What proves that the change succeeded
One green signal rarely covers the whole result. Each check has a different responsibility.
The file contains the expected data
A checksum can prove that the expected bytes are on disk. That is useful both after apply and after recovery.
But matching checksums do not prove that the running kubelet has read that file.
A mismatch and an inability to check must also remain distinct. A file-read error or command failure is not an empty checksum. It means there is no reliable result.
The service is running
systemctl is-active kubelet checks the service state from systemd’s point of view.3 That is a necessary signal, but it does not verify specific kubelet settings or prove stable operation.
A configuration change should also prove that the expected restart actually happened instead of merely observing a process that was already running. It should also detect a service that is repeatedly restarting.
The node is available to the cluster
Node Ready is another signal. It describes node health, but it does not prove that the intended configuration revision is active.
Freshness matters. The Node status and its Lease are updated independently, so a previously stored Ready=True value cannot by itself prove recovery after a restart that just happened.4
A practical success criterion should use observations made after the change and require some period of stable behavior. The duration belongs to the specific operation rather than to a universal recipe.
Kubelet is using the expected settings
Kubernetes documents the /configz endpoint for inspecting the effective configuration of a running kubelet.1 It helps verify process settings rather than only the file on disk.
The comparison should focus on managed fields while accounting for defaults and overrides.
Access to that check needs to be designed in advance. Broad nodes/proxy permission should not be treated as harmless read-only access: it exposes the kubelet API and can allow actions against containers.5
If the required verification cannot be performed, the result remains unconfirmed. An indirect green signal should not silently turn that into success.
Rollback needs its own verification
Apply answers one question:
Did the node move to the target configuration and preserve the required behavior?
Rollback answers another:
Did the node return to the selected recovery point, and is that recovery confirmed?
Restoring the file answers only part of the second question.
Recovery still needs to verify data integrity, activate the restored configuration when necessary, confirm kubelet operation, and inspect its settings. The categories of checks stay similar, but the expected state is the recovery point rather than the failed target configuration.
It is useful to distinguish three outcomes:
| Outcome | What is known |
|---|---|
| Apply failed, rollback confirmed | The target configuration was rejected and the node was restored |
| Apply failed, rollback failed | Recovery failure is confirmed |
| Apply failed, rollback outcome unknown | There is not enough evidence to determine the node state |
The third outcome matters when connectivity is lost or a check times out. No response proves neither that a command ran nor that it did not.
A successful rollback command may return exit code 0. That still does not make the original change successful: its target was not reached. Both outcomes need to remain visible in the report.
An operation failure is not always a recovery failure
There is another case that can disappear behind a generic failed status.
The restored configuration is already active. Kubelet checks passed. But the automation failed to persist the final operation state.
That is a failure and should not be hidden. But “rollback did not happen” would also be wrong: recovery happened, while recording the outcome failed.
This is why observed node state and operation execution state need to remain separate.
The same distinction affects retries. A later run should first determine what already happened instead of blindly copying files and restarting kubelet again. Otherwise, a reporting failure becomes unnecessary intervention on an already recovered node.
For the same reason, repeating rollback is not automatically safe just because it restores the same file again. Another process may have changed the configuration, and another restart may no longer be appropriate.
Who performs rollback when kubelet is down
A recovery mechanism has to be checked together with its dependencies.
If the automation relies on kubectl exec, that path depends on kubelet availability. Kubernetes uses API server-to-kubelet connections for operations against running containers.6 A kubelet failure can therefore remove the same execution path the recovery tool normally uses.
This does not mean every automation system must run outside Kubernetes. It means every claimed failure scenario needs an answer to one question: does an executor remain available that can restore the configuration and start the service?
That executor may be a pre-existing local mechanism or a separate administrative channel. Its availability has to be tested specifically while kubelet is unavailable.
This is also where manual and automatic rollback diverge. Having a rollback command does not mean it will automatically run after a failed apply. Automatic rollback needs its own trigger conditions, limits, and stop rules.
What configuration rollback does not promise
Restoring YAML is not the same as restoring the whole node to an earlier point in time.
Some settings have additional state. Changing the CPU Manager policy, for example, requires a separate procedure that also deals with the manager’s persisted state.7 It is not just a one-line configuration edit.
A rollback mechanism therefore needs an explicit boundary around the kinds of changes it supports. Anything that requires state migration, workload preparation, or extra recovery actions should not be hidden behind the promise “we can restore the config.”
Recovering kubelet also does not automatically mean the node should accept new workloads. uncordon is a separate action that makes the node schedulable again.8 It should happen only after the required checks and with awareness of why the node was taken out of scheduling in the first place.
The failed target configuration also needs a defined fate. If the next automation cycle applies it again, recovery becomes only a pause between identical failures.
A failed revision should remain blocked until a separate decision is made instead of receiving another attempt simply because the node is reachable again.
What to test
The useful tests are not limited to the happy path. They stop execution at the boundaries between actions.
| Scenario | Expected behavior |
|---|---|
| Backup is missing or corrupted | Recovery does not start from untrusted data |
| Execution stops after file replacement | The next run recognizes an incomplete operation |
| File is restored but kubelet does not start | File recovery is not reported as successful rollback |
| Verification times out | The result is marked as unconfirmed |
| Node is recovered but final status cannot be persisted | Node state and reporting failure remain distinct |
| Rollback runs again or overlaps with another change | Newer state is not blindly overwritten |
These are criteria for testing a recovery mechanism, not a claim that every implementation already satisfies them.
Conclusion
A backup answers “what can we restore?” A restart command answers “how can we try to activate it?”
A rollback contract has to answer more: why this recovery point is valid, who can restore it when kubelet is unavailable, what evidence confirms the result, and what happens after partial execution.
Rollback is complete not when the tool runs its last command, but when the node state is known with the required confidence — or when the system records honestly that it could not determine that state.