EMFILE too many open files on fs.watch (Node.js) as root on VPS with 100+ Docker containers
Diagnosis
fs.inotify.max_user_instances kernel limit (default 128) is exhausted by root. Each containerd-shim process consumes 1 inotify FD per Docker container (~100+ on this VPS), plus systemd, traefik, cadvisor, fail2ban, udisksd consume the rest. Root hits the 128/128 ceiling before any user-space file watcher can start. Raising LimitNOFILE in the systemd unit does NOT fix it — the limit is per-user-id inotify instances, not per-process file descriptors. Error message is misleading (says “too many open files” = EMFILE, but actual syscall is inotify_init1). Verification: for d in /proc/*/fd; do ...; done | grep anon_inode:inotify shows exact count per UID.
Fix
Create /etc/sysctl.d/99-<service>-inotify.conf with fs.inotify.max_user_instances=512 (4x default). Apply with sysctl -p /etc/sysctl.d/99-<service>-inotify.conf. Persistent across reboots. Doesn’t require per-service tuning — raises the root UID ceiling for all file-watcher services.
Related
- docker
- 2026-04-04-oracle-001-self-architecture-analysis
- inotify-limit-saturation-from-docker-containers
- inotify-limit-saturated-by-docker-container-shims
- graphiti
- inotify-max-user-instances-exhausted-by-docker-shims