Too many open files

Too many open files is EMFILE: a process asked for another file descriptor and was refused because it had reached its limit. On a server this usually means sockets, not files. Every connection is a descriptor, so a busy proxy or a service with a leaking connection pool hits it long before anything involving actual files.

Find the real limit

The single most common mistake here is checking the wrong limit.

ulimit -n

That is your shell's limit. A daemon started by systemd never saw your shell and does not inherit it. Read the process itself:

pid=$(pgrep -n nginx)
cat /proc/$pid/limits | grep "open files"

Max open files            1024                 524288               files

Soft limit 1024, hard limit 524288. The soft limit is what applies; the hard limit is the ceiling a process may raise itself to.

Count what it is using

ls /proc/$pid/fd | wc -l

Sitting just under the soft limit confirms the diagnosis. Well below it means something else produced the error, most likely a different process or a system-wide limit.

To see what the descriptors are:

ls -l /proc/$pid/fd | awk '{print $NF}' | sed 's/:.*//' | sort | uniq -c | sort -rn | head

   3891 socket
     42 /var/log/app/access.log
      8 /dev/null

Nearly four thousand sockets is either legitimate load or a leak. Distinguish them:

ss -tanp | grep "pid=$pid" | awk '{print $1}' | sort | uniq -c

   3784 CLOSE-WAIT
    107 ESTAB

A large CLOSE-WAIT count is a leak, and a specific one: the peer closed the connection and your application never called close(). No limit increase fixes that, it only postpones it. Look for an HTTP client whose response bodies are not being closed, or a connection pool with no idle eviction.

Find the worst offenders

for p in /proc/[0-9]*; do
  echo "$(ls $p/fd 2>/dev/null | wc -l) $(cat $p/comm 2>/dev/null)"
done | sort -rn | head -10

Faster than lsof on a busy box, and it does not need root for your own processes. lsof -n | awk '{print $1}' | sort | uniq -c | sort -rn | head gives the same answer if you prefer it.

Raise it: systemd services

This is where most of the confusion lives. For a systemd service, /etc/security/limits.conf is ignored. That file is read by PAM, and systemd does not start services through PAM.

sudo systemctl edit nginx

[Service]
LimitNOFILE=65535

sudo systemctl daemon-reload
sudo systemctl restart nginx
systemctl show nginx -p LimitNOFILE

systemctl edit creates a drop-in under /etc/systemd/system/nginx.service.d/, which survives package upgrades. Editing the unit file in /lib/systemd/system/ does not.

A restart is required. systemctl reload does not change resource limits, because they are set at process creation.

Set a default for all services in /etc/systemd/system.conf:

[Manager]
DefaultLimitNOFILE=65535

Raise it: interactive shells

For logins, limits.conf is the right place:

# /etc/security/limits.conf
*    soft    nofile    65535
*    hard    nofile    65535

Requires pam_limits.so in the PAM stack, which is standard on most distributions, and takes effect on the next login rather than immediately.

Containers

Docker:

docker run --ulimit nofile=65535:65535 myapp:1.0

# compose.yaml
services:
  app:
    ulimits:
      nofile:
        soft: 65535
        hard: 65535

Kubernetes has no field for this. There is no ulimits in a Pod spec. The container inherits the limit from the container runtime's configuration on the node, so you change it there:

// /etc/docker/daemon.json
{ "default-ulimits": { "nofile": { "Name": "nofile", "Soft": 65535, "Hard": 65535 } } }

For containerd, the equivalent lives in the runtime config or the systemd unit for containerd itself. On a managed cluster this is a node-pool or launch-template change rather than something you can do from a manifest.

The system-wide ceiling

Per-process limits sit under a system-wide one:

cat /proc/sys/fs/file-max
cat /proc/sys/fs/file-nr        # allocated, unused, max

Raising a per-process limit above file-max achieves nothing. On modern kernels file-max is derived from memory and is usually large, so this is rarely the binding constraint, but it is worth ruling out.

# /etc/sysctl.d/99-files.conf
fs.file-max = 2097152

sudo sysctl --system

Note that inotify watches have their own separate limit and produce a similar-sounding failure. If the error mentions inotify, raise fs.inotify.max_user_watches instead.

Decide: leak or load?

Before raising anything, answer this. A limit increase on a leaking process buys you hours and then the same outage, usually at a worse time.

watch -n5 "ls /proc/$pid/fd | wc -l"

Under steady traffic, a count that climbs and never falls is a leak. A count that rises and falls with load is capacity, and a higher limit is the correct fix.

A checklist

1. cat /proc/PID/limits, not ulimit -n.
2. ls /proc/PID/fd | wc -l and compare.
3. Break down the descriptors. Mostly sockets → look at connection handling.
4. ss -tanp | grep pid=PID. Lots of CLOSE-WAIT → a leak, not a limit.
5. watch the count. Climbing under steady load → a leak.
6. systemd service → systemctl edit, LimitNOFILE=, daemon-reload, restart.
7. Interactive shell → /etc/security/limits.conf, effective next login.
8. Kubernetes → no Pod field exists; change it on the node runtime.

---

Frequently Asked Questions

Why does ulimit -n show a high limit but my service still fails?

Because ulimit -n reports the limit of your interactive shell, and a daemon started by systemd never inherited it. systemd sets resource limits itself from the unit file and does not consult your shell or /etc/security/limits.conf, which is read by PAM at login. Read the actual value with cat /proc/PID/limits, and change it with systemctl edit <service> adding LimitNOFILE=, then daemon-reload and restart. A reload is not enough, because limits are applied when the process is created.

Should I just raise the limit?

Only once you know whether it is load or a leak. Watch the descriptor count with watch -n5 "ls /proc/PID/fd | wc -l" under steady traffic: rising and falling with load means you need more headroom, and climbing monotonically means the process is not closing things. For a leak, a higher limit buys hours and then produces the same outage later, usually at a worse moment. A large number of sockets in CLOSE-WAIT, which ss -tanp will show you, is the clearest leak signature there is.

Why does /etc/security/limits.conf not work for my daemon?

That file is applied by the pam_limits PAM module during login, and systemd does not start services through PAM. Anything launched as a systemd unit gets its limits from the unit file, falling back to DefaultLimitNOFILE in /etc/systemd/system.conf. limits.conf still governs interactive shells and anything started from one, which is exactly why a service behaves differently when you run it by hand in a terminal than when systemd starts it at boot.

How do I set the open file limit in Kubernetes?

There is no Pod or container field for it. The limit comes from the container runtime on the node, so you set it in the runtime's configuration, for example default-ulimits in /etc/docker/daemon.json or the containerd equivalent, and restart the runtime. On a managed cluster that means a node-pool image, launch template or startup script rather than anything in a manifest. A privileged init container calling ulimit is sometimes suggested and does not work, because the limit applies to that container's process and not to the others in the Pod.

What is the difference between the soft and hard limit?

The soft limit is enforced, and the hard limit is the ceiling the soft limit may be raised to. An unprivileged process can raise its own soft limit up to the hard limit but cannot exceed it; only a privileged process can raise the hard limit. /proc/PID/limits shows both. Some servers, including nginx with worker_rlimit_nofile, raise their own soft limit at startup, which is why setting a generous hard limit in the unit file is worth doing even when the soft limit looks adequate.