OOMKilled (Exit Code 137)

OOMKilled means the Linux kernel's out-of-memory killer terminated your container because it exceeded its memory limit.

The defining characteristic: SIGKILL cannot be caught, so the process dies instantly with no chance to log, flush or clean up. That is why the application's own logs simply stop mid-sentence and explain nothing.

Confirm it

kubectl get pod payments-api-7d4f-x8k2 \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'
# OOMKilled

kubectl describe pod payments-api-7d4f-x8k2

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Restart Count:  4

A Pod that restarts periodically with a clean log and no application error is nearly always this. Exit code 137 is 128 + 9, meaning SIGKILL.

Container OOM is not node OOM

Two different events, frequently confused, with different fixes.

# Evictions, which are the node-level case
kubectl get pods --field-selector status.phase=Failed
kubectl describe node <node> | grep -A5 Conditions

If Pods are being evicted rather than OOMKilled, the node is oversubscribed — the answer is right-sizing requests or adding capacity, not raising one container's limit.

Is it a bad limit, or a leak?

This is the question that decides the fix, and raising the limit before answering it just moves the crash later.

kubectl top pod payments-api-7d4f-x8k2 --containers

A single reading is not enough. What you want is the shape over time:

# Steady, near the limit  → limit is too low
# Sawtooth, resets on restart → fine, sized tightly
# Monotonic climb until death → leak. A bigger limit only delays it.

If you have Prometheus, container_memory_working_set_bytes over a few days answers it immediately. Without it, watch kubectl top across several restart cycles — a leak shows as a climb that always ends at the same ceiling regardless of what that ceiling is.

A limit raised twice that is OOMKilled twice is a leak. Stop raising it.

The runtime heap trap

By far the most common non-obvious cause, and it affects the JVM, Node.js and .NET.

These runtimes size their heap against what they believe is available memory. Without being told about the cgroup limit, older versions read the node's total — so a container limited to 512Mi may size a heap for a 64GB machine and get killed the moment it grows.

JVM:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75.0"

Modern JVMs are container-aware and respect cgroup limits, but MaxRAMPercentage makes the headroom explicit. Use the percentage rather than a fixed -Xmx so the setting survives a limit change.

Crucially, heap is not the whole footprint. Metaspace, thread stacks, direct byte buffers and the JIT code cache all live outside it, which is why a container with -Xmx512m and a 512Mi limit is OOMKilled reliably. Leave 25–30% for non-heap.

Node.js:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=384"      # for a 512Mi limit

.NET: respects limits when DOTNET_GCHeapHardLimit or the container limit is detected, but verify rather than assume on older versions.

Sizing it properly

resources:
  requests:
    memory: 256Mi     # steady-state usage, scheduler reserves this
  limits:
    memory: 512Mi     # real peak plus headroom

Three things worth getting right:

Set requests from observed steady state, plus roughly 25%. Requests drive scheduling; too high wastes cluster capacity, too low means the Pod is placed on a node with no room to grow.

Set the limit above real peak, not average. Traffic spikes, garbage collection cycles and batch operations all produce transient peaks that a limit sized to the average will kill.

Consider whether the limit should exist at all for a trusted workload on a dedicated node. Without one the container can use whatever the node has — which risks the node rather than the Pod, so this is a deliberate trade and not a default.

Note that memory limits kill while CPU limits throttle. That asymmetry is why memory limits matter so much more: exceeding CPU makes you slow, exceeding memory makes you dead.

Not enough memory anywhere?

If the workload genuinely needs more than fits, the options are a larger instance type, a memory-optimised family such as AWS r instances, or splitting the work. A Vertical Pod Autoscaler in recommendation mode will tell you what it actually wants based on real usage rather than guesswork.

A checklist

1. lastState.terminated.reason — confirm it says OOMKilled and not something else.
2. Check whether Pods are Evicted instead, which is a node problem, not a limit problem.
3. kubectl top pod --containers over several cycles — steady, sawtooth, or climbing?
4. Climbing to the ceiling every time → it is a leak. Fix the application.
5. Steady near the limit → raise the limit above real peak.
6. JVM, Node or .NET → set MaxRAMPercentage or --max-old-space-size, leaving non-heap headroom.
7. Re-check after the change; an OOM that recurs at a higher limit was never a sizing problem.

---

Frequently Asked Questions

What does OOMKilled mean in Kubernetes?

The container exceeded its memory limit and the Linux kernel's out-of-memory killer terminated it with SIGKILL. Because SIGKILL cannot be caught or handled, the process dies immediately with no opportunity to log anything — which is why the application's output simply stops with no error. Confirm it by reading lastState.terminated.reason, which returns OOMKilled, and the exit code, which will be 137.

What is exit code 137?

128 plus 9, where 9 is SIGKILL. In Kubernetes there are two realistic sources: the kernel OOM killer when a container exceeds its memory limit, and the kubelet terminating a container that failed its liveness probe. The termination reason distinguishes them — OOMKilled for the first, Error accompanied by a Killing event for the second. Exit code 143 is 128 plus 15, SIGTERM, which usually means an ordinary shutdown rather than a fault.

Should I just increase the memory limit?

Only after establishing that the limit is genuinely too low rather than the application leaking. Watch kubectl top pod --containers across several restart cycles: usage that sits steadily near the limit means the limit is too small, while usage that climbs monotonically until death means a leak, and a bigger limit only postpones the crash. If you have raised the limit twice and been OOMKilled twice, stop raising it and investigate the application.

Why does my Java application get OOMKilled when the heap is smaller than the limit?

Because heap is not the whole memory footprint. Metaspace, thread stacks, direct byte buffers, the JIT code cache and the JVM's own overhead all live outside the heap, so a container with -Xmx512m and a 512Mi limit will be killed. Use -XX:MaxRAMPercentage=75 rather than a fixed -Xmx, which reserves roughly a quarter of the limit for non-heap usage and automatically adapts if you change the limit later.

What is the difference between OOMKilled and an evicted pod?

OOMKilled is per-container: that container exceeded its own limit and the kernel killed it, while the Pod stays Running with an incremented restart count. Eviction is per-node: the node itself ran short of memory and the kubelet removed Pods to reclaim it, choosing BestEffort first, then Burstable, then Guaranteed. An evicted Pod shows status Failed with reason Evicted. The fixes differ — one is a limit problem, the other is a capacity or requests problem.

How can I see memory usage before a container is killed?

kubectl top pod --containers gives a current snapshot but requires metrics-server and shows nothing about history — and a container killed thirty seconds ago leaves no reading. Prometheus recording container_memory_working_set_bytes is what actually answers the question, since it shows the shape over time and distinguishes a steady high-water mark from a monotonic climb. Vertical Pod Autoscaler in recommendation mode uses the same data to suggest limits without changing anything.