Cannot allocate memory from fork() is ENOMEM, and the kernel returns it for several reasons that are not "the machine is out of RAM". Establish which before changing anything.
Is memory actually the problem?
free -h
total used free shared buff/cache available
Mem: 31Gi 8Gi 1Gi 0.5Gi 22Gi 22Gi
Read available, not free. Page cache in buff/cache is reclaimable, so 1Gi free with 22Gi available is a healthy machine. Only a low available means genuine memory pressure.
With 22Gi available and forks failing, memory is not the constraint.
Count threads, not processes
ps -eLf | wc -l # the L is threads
cat /proc/sys/kernel/pid_max
cat /proc/sys/kernel/threads-max
Every thread consumes a PID from the same space as processes. A JVM or a Go service with thousands of threads exhausts pid_max long before it exhausts memory, and java.lang.OutOfMemoryError: unable to create new native thread is the same failure surfaced by the JVM.
Find the biggest consumers:
ps -eLo pid,nlwp,comm --sort=-nlwp | head -10
PID NLWP COMMAND
4821 2048 java
9134 512 myservice
If one process holds thousands of threads, that is a leak worth fixing rather than a limit to raise.
Raise the limits if the count is legitimate:
# /etc/sysctl.d/99-pids.conf
kernel.pid_max = 4194304
kernel.threads-max = 4194304
sudo sysctl --system
4194304 is the current maximum on 64-bit Linux.
The cgroup limit
This is the one people miss, because the host looks fine.
systemd sets TasksMax on every unit, defaulting to 15% of kernel.pid_max:
systemctl show myservice -p TasksMax -p TasksCurrent
TasksMax=4915
TasksCurrent=4915
At the limit, the service cannot fork while the host has capacity to spare.
sudo systemctl edit myservice
[Service]
TasksMax=16384
sudo systemctl daemon-reload
sudo systemctl restart myservice
TasksMax=infinity removes the limit, and removes the protection against a fork bomb taking the machine down with it.
In a container the same limit is pids.max:
docker run --pids-limit 4096 myapp:1.0
Kubernetes has no per-Pod field. The kubelet's podPidsLimit sets it node-wide, defaulting to 4096 on recent versions:
# kubelet config
podPidsLimit: 8192
Inside a container, check what you are actually subject to:
cat /sys/fs/cgroup/pids.max # cgroup v2
cat /sys/fs/cgroup/pids/pids.max # cgroup v1
RLIMIT_NPROC
A per-user limit, separate from everything above:
ulimit -u
cat /proc/$$/limits | grep processes
nproc counts every process for that uid across the whole system, not per session. A service account running many workers hits it while other users are unaffected.
# /etc/security/limits.conf
appuser soft nproc 16384
appuser hard nproc 32768
As ever, this does not apply to systemd services. Those use LimitNPROC= in the unit.
Overcommit
cat /proc/sys/vm/overcommit_memory
| Value | Behaviour |
|---|---|
0 | Heuristic. The default |
1 | Always allow |
2 | Strict: refuse beyond CommitLimit |
With 2, allocations are refused once Committed_AS reaches CommitLimit, which produces ENOMEM while free -h shows memory available:
grep -E "Committed_AS|CommitLimit" /proc/meminfo
Mode 2 is a deliberate choice for predictability, at the cost of refusing allocations processes would never have touched. fork() is particularly affected, because it nominally duplicates the parent's address space even though copy-on-write means it rarely uses it. A 16GB process forking to run a 1KB shell script needs 16GB of commit under strict accounting, which is why large JVMs and databases hit this.
That specific case is what vm.overcommit_ratio and, for databases, posix_spawn instead of fork exist to address.
When it is genuinely memory
grep -i "killed process" /var/log/syslog
sudo journalctl -k | grep -i "out of memory"
An OOM killer entry names the process and its score. In a container, OOMKilled and exit code 137 are the equivalent signals.
sudo dmesg -T | grep -A5 "Out of memory"
A checklist
free -h, readingavailablerather thanfree.ps -eLf | wc -lagainstpid_max. Threads count.ps -eLo pid,nlwp,comm --sort=-nlwpto find a thread leak.systemctl show <unit> -p TasksMax -p TasksCurrent. The default is 15% ofpid_max.- Container →
cat /sys/fs/cgroup/pids.max, and--pids-limitorpodPidsLimit. ulimit -u, which counts across the whole system for that uid.vm.overcommit_memory=2refuses allocations well before memory runs out.- Genuine exhaustion → the OOM killer leaves a record in the kernel log.
Frequently Asked Questions
Why does fork fail when free shows plenty of memory?
Because ENOMEM from fork() covers several limits that are not memory. The most common are the PID space, which threads consume as well as processes, and cgroup pids.max, which systemd sets per unit at 15% of the system maximum by default. Strict overcommit accounting can also refuse a fork while memory is free, because fork() nominally duplicates the parent's entire address space. Check the thread count and the cgroup limit before concluding you need more RAM.
What is the difference between free and available in free -h?
free is memory doing nothing at all; available is memory the kernel can give to a new allocation, which includes reclaimable page cache. A healthy Linux machine deliberately uses most of its RAM for cache, so free being small is normal and not a problem. available is the number to watch, and only a low value indicates real pressure. Monitoring or alerting built on free rather than available produces constant false alarms.
Why does my service hit a limit when the host has capacity?
Because it is confined by its own cgroup. systemd applies TasksMax to every unit, defaulting to 15% of kernel.pid_max, so a service can exhaust its allowance on a machine with plenty of headroom. systemctl show <unit> -p TasksMax -p TasksCurrent shows both numbers. Raise it with a drop-in via systemctl edit, and prefer a specific larger value over infinity, which also removes the protection that stops a runaway process taking the whole machine down.
How do I set a PID limit for a Kubernetes Pod?
There is no per-Pod field. The kubelet applies podPidsLimit to every Pod on the node, defaulting to 4096 on recent versions, and changing it is a node configuration change rather than something you can express in a manifest. Inside a container, cat /sys/fs/cgroup/pids.max shows the limit you are actually subject to. If one workload genuinely needs far more threads than the rest, a dedicated node pool with a higher kubelet setting is the usual way to accommodate it.
Should I set vm.overcommit_memory to 2?
Only deliberately, and knowing the cost. Mode 2 refuses allocations once committed memory reaches CommitLimit, which gives predictable behaviour and avoids the OOM killer, but it accounts for memory processes may never touch. fork() suffers most, because it nominally needs a copy of the parent's whole address space even though copy-on-write means it rarely uses any of it, so a large JVM or database forking a small child can fail outright. The default heuristic mode is the right choice for most systems.