Problem

After identifying the inotify watchers limit issue, the next question was straightforward: which processes were actually consuming those watchers?

Understanding the consumers turned out to be more interesting than hitting the limit itself.

Symptoms

By default, Linux only shows the configured limit.

It does not show which processes are responsible for consuming inotify watchers.

As a result, troubleshooting can quickly turn into guesswork.

Investigation

Several observations became obvious during the investigation.

inotify watchers consumption is rarely caused by a single process.

Instead, it is usually distributed across many different components.

The most common contributors are:

  • container runtimes;
  • log collectors;
  • monitoring systems;
  • service meshes;
  • applications watching configuration files;
  • Kubernetes operators.

Individually, each component looks harmless.

Together, they can gradually become a problem.

Another observation is the growth pattern.

Unlike CPU spikes or memory leaks, inotify watchers consumption can increase for weeks or months without triggering any alerts.

Nothing appears broken until an unrelated operation suddenly starts behaving unexpectedly.

Takeaways

Capacity planning should not be limited to Kubernetes resources alone.

Teams usually focus on:

  • CPU;
  • memory;
  • disks;
  • pod density per node.

Linux system limits deserve the same level of attention.

fs.inotify.max_user_watches should be treated as another infrastructure resource instead of a value that remains untouched forever.

Long-term visibility is equally important.

Knowing the current value is useful.

Knowing which processes steadily increase inotify watchers consumption over time is far more valuable.