Ephemeral storage is the node's local disk used by Pods: container writable layers, emptyDir volumes, and logs written to stdout. It is not a PersistentVolume. When it runs short, the kubelet evicts Pods to get it back.
Two distinct mechanisms produce an eviction, and they need different fixes.
Which one was it?
kubectl describe pod my-pod | grep -A5 -i evict
Your own limit.
Pod ephemeral local storage usage exceeds the total limit of containers 1Gi
You set limits.ephemeral-storage and the Pod exceeded it. This is your Pod's fault and only your Pod is affected.
The node's threshold.
The node was low on resource: ephemeral-storage.
Container app was using 4521032Ki, which exceeds its request of 0.
The node hit DiskPressure and the kubelet picked victims. Note the phrase exceeds its request of 0: with no requests.ephemeral-storage set, any usage exceeds the request, which makes the Pod a preferred eviction candidate. Pods that declare a request and stay under it are evicted last.
kubectl get nodes -o custom-columns=NAME:.metadata.name,DISK:.status.conditions[?\(@.type==\"DiskPressure\"\)].status
Where the disk went
On the node, four things consume it:
| Consumer | Path |
|---|---|
| Image layers | /var/lib/containerd or /var/lib/docker |
| Container writable layers | same |
| Container logs | /var/log/pods |
emptyDir volumes | /var/lib/kubelet/pods |
kubectl debug node/ip-10-0-1-42 -it --image=busybox -- sh
# then, inside:
du -sh /host/var/lib/containerd /host/var/log/pods /host/var/lib/kubelet 2>/dev/null
kubectl debug node mounts the host filesystem under /host, which is the supported way in without SSH.
In practice it is nearly always image layers on a long-lived node, or one Pod writing an enormous log.
Fix 1: Set requests and limits
resources:
requests:
ephemeral-storage: "1Gi"
limits:
ephemeral-storage: "4Gi"
The request protects you: the scheduler reserves it, and eviction ranking puts Pods under their request last. The limit protects the node from you.
The tradeoff is real. A Pod that exceeds its ephemeral-storage limit is evicted immediately with no grace, so set the limit above your genuine peak. A batch job that writes a large temporary file needs headroom for it.
Fix 2: Cap emptyDir
An emptyDir with no sizeLimit can consume the whole node.
volumes:
- name: scratch
emptyDir:
sizeLimit: 2Gi
For caches and scratch space that fit comfortably in RAM, a memory-backed emptyDir avoids disk entirely:
emptyDir:
medium: Memory
sizeLimit: 512Mi
Be careful: memory-backed emptyDir counts against the container's memory limit, so writing 512Mi into it with a 256Mi memory limit gets you OOMKilled rather than a disk eviction. Raise the memory limit accordingly.
Fix 3: Logs
Everything a container writes to stdout is stored on the node. A debug-level log in production fills a disk quickly, and the default rotation is more generous than people expect.
The kubelet rotates container logs with containerLogMaxSize (default 10Mi) and containerLogMaxFiles (default 5), so 50Mi per container. With 40 Pods on a node that is 2Gi before anything else.
# /var/lib/kubelet/config.yaml
containerLogMaxSize: 10Mi
containerLogMaxFiles: 3
The application-side fix is better: log at info in production, and write large diagnostic output to an object store rather than stdout.
Fix 4: Image garbage collection
The kubelet removes unused images only when disk usage crosses a threshold:
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
At 85% full it deletes unused images until it reaches 80%. On a node that accumulates many image versions, lowering these reclaims space earlier:
imageGCHighThresholdPercent: 70
imageGCLowThresholdPercent: 60
Note that the eviction threshold defaults to nodefs.available<10%, which is above the image GC high threshold in disk-full terms. So a node can start evicting Pods while image GC has already run and found nothing more to delete. If that is your pattern, the node's disk is genuinely too small.
Clean up the evicted Pods
Evicted Pods stay in the API as a record. They consume no resources but clutter everything:
kubectl get pods --all-namespaces --field-selector status.phase=Failed
kubectl delete pods --all-namespaces --field-selector status.phase=Failed
The kubelet's --pod-eviction-timeout and the controller's terminated-pod GC handle this eventually, at a threshold of 12,500 Pods by default, which is why they appear to accumulate forever on a small cluster.
A checklist
kubectl describe podand read which mechanism evicted it.exceeds the total limit of containers→ your limit. Raise it or write less.node was low on resource→ the node. Your Pod may be collateral damage.exceeds its request of 0→ setrequests.ephemeral-storageso you rank later for eviction.kubectl debug node/<node>anddu -shthe four paths.- Add
sizeLimitto everyemptyDir. - Check log volume. Default rotation allows 50Mi per container.
- Tune
imageGCHighThresholdPercentdown on nodes that hoard images.
Frequently Asked Questions
What counts as ephemeral storage in Kubernetes?
The node's local disk used by Pods: each container's writable layer, emptyDir volumes, and anything written to stdout or stderr, which the kubelet stores as files under /var/log/pods. It explicitly does not include PersistentVolumes, which are separate and not subject to eviction. Image layers live on the same filesystem and count towards the node filling up, though they are not attributed to any individual Pod, which is why a node can hit DiskPressure when no single Pod is using much.
Why was my Pod evicted when a different Pod filled the disk?
Because node-level eviction is about reclaiming space, not about blame. When the kubelet hits its eviction threshold it ranks Pods by how far their usage exceeds their request and by QoS class, then evicts from the top. A Pod with no requests.ephemeral-storage exceeds its request of zero by definition, so it ranks as a candidate regardless of whether it used much. Setting a realistic request is the single most effective way to avoid being chosen as the victim.
How do I limit how much disk a Pod can use?
Set limits.ephemeral-storage in the container's resources, and sizeLimit on any emptyDir volume. The limit is enforced strictly: exceeding it evicts the Pod immediately with no grace period, so set it above your real peak rather than at it. Also set requests.ephemeral-storage, which both reserves capacity at scheduling time and improves your position in the eviction ranking when the node comes under pressure for reasons that have nothing to do with you.
Does a memory-backed emptyDir avoid the disk problem?
It avoids the disk, and moves the cost to memory. emptyDir with medium: Memory is a tmpfs, so it never touches the node's disk, but its contents count against the container's memory limit. Writing 512Mi into one with a 256Mi memory limit gets the container OOMKilled instead of evicted, which is a worse failure because it is less obviously related to the volume. It is a good option for small caches and scratch space, provided you raise the memory limit to cover the sizeLimit.
Why does the node still run out of disk after image garbage collection?
Because the kubelet's image GC high threshold, 85% by default, is less aggressive than the eviction threshold, nodefs.available<10%. Image GC only removes images not used by any container, so a node running many distinct images has little it can reclaim. If GC has run and the node still evicts, the honest conclusion is usually that the node's disk is too small for the workload density. Lowering imageGCHighThresholdPercent helps nodes that accumulate many versions of the same image over time.