Kubernetes Resource Management

Two numbers per container decide almost everything about how your workload behaves under pressure: requests, which drive scheduling, and limits, which cap consumption at runtime.

Get them wrong and you get nodes that look idle while Pods cannot schedule, containers killed at 2am, or latency that appears only under load and makes no sense from the application's own metrics.

Requests and limits are two different mechanisms

The scheduler only ever looks at requests. It sums the requests of everything already on a node and places your Pod where the remaining capacity fits. It never looks at actual usage.

That is why a node can show 20% real CPU usage in monitoring and still refuse to schedule anything. Its requests are fully committed. kubectl describe node shows both, and the gap between them is usually the story:

kubectl describe node ip-10-0-1-42 | grep -A8 "Allocated resources"

Allocated resources:
  Resource   Requests      Limits
  cpu        3800m (95%)   7200m (180%)
  memory     6.1Gi (81%)   12Gi (160%)

Limits exceeding 100% is normal, that is deliberate overcommit, betting that not everything peaks simultaneously. Requests at 95% means this node is effectively full.

Units

CPU is measured in cores, and m means millicores. Thousandths of a core.

1     = 1 core
500m  = half a core
100m  = a tenth of a core

CPU is compressible: exceed your limit and you are throttled, not killed.

Memory is measured in bytes with suffixes. The distinction between the two families catches people:

128Mi = 128 × 1024² = 134,217,728 bytes   (mebibytes)
128M  = 128 × 1000² = 128,000,000 bytes   (megabytes)

Use Mi and Gi. Memory is incompressible: exceed the limit and the kernel kills the process. There is no throttling option.

What actually happens at the limit

Exceeding a CPU limit throttles the container. The kernel simply stops scheduling it for the remainder of each 100ms period. Nothing crashes; requests just get slower. This is genuinely hard to diagnose from inside the application, because the process sees time passing with no work done.

kubectl exec -it <pod> -- cat /sys/fs/cgroup/cpu.stat | grep throttled

A high nr_throttled relative to nr_periods means the limit is costing you latency.

Exceeding a memory limit gets the container OOMKilled immediately.

kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

OOMKilled

A Pod that restarts periodically with no error in its own logs is the signature. The application is killed without warning, so it never gets to log anything.

The CPU limit argument

Setting CPU requests is uncontroversial. Do it always.

Setting CPU limits is genuinely contested. The case against: throttling adds latency, and multi-threaded runtimes hit the limit sooner than intuition suggests because the quota is consumed across all threads. A service limited to 1000m running eight threads can exhaust its quota in 12ms of wall-clock time and then stall for 88ms.

The case for: without limits, one badly behaved workload starves its neighbours, and performance becomes unpredictable in a way that depends on what else happens to be scheduled nearby.

A defensible middle position: always set memory limits, an unbounded leak takes down the whole node, and set CPU requests but be cautious with CPU limits, particularly on latency-sensitive services. If you do set CPU limits, set them well above observed peak, not at it.

QoS classes

Kubernetes assigns each Pod a class from its requests and limits, and that class decides eviction order when a node runs out of memory.

kubectl get pod <name> -o jsonpath='{.status.qosClass}'

BestEffort Pods are evicted first and are also placed carelessly, since the scheduler thinks they need nothing. A production workload with no requests set is not merely unoptimised. It is first in line to be killed under pressure.

For anything critical, Guaranteed gives the strongest protection: set requests equal to limits.

Namespace-level controls

LimitRange supplies defaults and bounds within a namespace, so a Pod submitted with nothing set does not become BestEffort:

apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      default:              # applied as the limit if unset
        cpu: 500m
        memory: 512Mi
      defaultRequest:       # applied as the request if unset
        cpu: 100m
        memory: 128Mi
      max:
        cpu: "4"
        memory: 8Gi

ResourceQuota caps the namespace in aggregate, which is how you stop one team consuming a shared cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"

One consequence worth knowing: once a ResourceQuota constrains a resource in a namespace, every Pod must specify it. A Pod with no memory request is rejected outright. Pair a ResourceQuota with a LimitRange so existing manifests keep working.

Choosing the numbers

Measure rather than guess:

kubectl top pods --containers
kubectl top nodes

A reasonable starting point:

• Memory request: observed steady-state usage plus roughly 25%.
• Memory limit: request plus headroom for real peaks. For JVM, Go or Node applications, make sure the runtime knows about the limit or it will size its heap against the node's total memory.
• CPU request: observed average, not peak.
• CPU limit: well above peak, or omitted for latency-sensitive services.

kubectl top needs metrics-server installed; without it the command fails. For real sizing you want historical data from Prometheus, since a single snapshot misses daily peaks.

Vertical Pod Autoscaler in recommendation mode is useful here: it observes actual usage and suggests values without changing anything.

Explore further in this section

Quick reference

---

Key takeaways

• The scheduler reads requests, never actual usage. A node at 20% CPU can still be unschedulable.
• Mi is not M. Use the binary suffixes.
• CPU throttles, memory kills. A silent periodic restart with clean logs is almost always OOMKilled.
• Always set memory limits. Be cautious with CPU limits on latency-sensitive services.
• No requests means BestEffort, which is scheduled carelessly and evicted first.
• Guaranteed, requests equal to limits, is the strongest protection for critical workloads.
• A ResourceQuota makes requests mandatory. Add a LimitRange alongside it or existing manifests break.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between requests and limits?

A request is what the scheduler reserves for a container when deciding which node can host it; a limit is the hard ceiling the kubelet and the kernel enforce at runtime. The scheduler only ever reads requests and never looks at actual usage, so requests determine placement and limits determine behaviour under pressure. Exceeding a CPU limit throttles the container; exceeding a memory limit kills it.

What does OOMKilled mean and how do I fix it?

The container exceeded its memory limit and the kernel terminated it. Confirm it with kubectl get pod <name> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'. The application gets no warning and so logs nothing, which is why a Pod that restarts periodically with clean logs is almost always this. Either the limit is genuinely too low, check real usage with kubectl top pods before raising it, or there is a leak that a higher limit only postpones. Runtimes with their own heap settings, the JVM especially, must be told about the container limit or they will size against the node's total memory.

Should I set CPU limits?

This is genuinely debated. CPU limits throttle rather than kill, and throttling adds latency that is difficult to diagnose because the application sees time passing with no work done. Multi-threaded runtimes hit the quota faster than expected, since it is consumed across all threads within each 100ms window. Without limits, though, one workload can starve its neighbours. A defensible position is to always set CPU requests, always set memory limits, and either omit CPU limits on latency-sensitive services or set them well above observed peak.

What are QoS classes and why do they matter?

Kubernetes derives a class from your requests and limits. Guaranteed means requests equal limits for every container and both resources; Burstable means requests are set but differ from limits; BestEffort means nothing is set. Under node memory pressure the kubelet evicts BestEffort first, then Burstable, then Guaranteed. A production workload with no requests is therefore first to be killed, as well as being placed by a scheduler that believes it needs nothing.

Why won't my Pod schedule when the node looks idle?

The scheduler works from the sum of requests on each node, not measured usage. If existing Pods request 95% of a node's allocatable CPU, nothing more fits there even if real usage is 20%. Run kubectl describe node <name> and look at the "Allocated resources" section, which shows requests and limits as percentages of capacity. Over-large requests are the most common cause of clusters that cost far more than their actual workload justifies.

How do I choose the right values?

Measure first. kubectl top pods --containers gives a current snapshot, and Prometheus history is better because a single reading misses daily peaks. A reasonable starting point is a memory request at observed steady-state plus about 25%, a memory limit with genuine headroom above that, and a CPU request at observed average rather than peak. Vertical Pod Autoscaler in recommendation mode watches real usage and suggests values without changing anything, which is a low-risk way to sanity-check your numbers.