A Pending Pod has been accepted by the API server but the scheduler cannot find a node for it. Unlike most Kubernetes failures, this one tells you exactly why, for every node.
The trick is knowing that the message is a summary of separate rejections, and the numbers add up to your node count.
Read the event
kubectl describe pod payments-api-7d4f-x8k2
Events:
Warning FailedScheduling 0/5 nodes are available:
2 Insufficient cpu,
2 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: },
1 node(s) had volume node affinity conflict.
That reads as: five nodes, two rejected for CPU, two are control-plane nodes, one has a volume in the wrong zone. Every node has a reason. Fix the reason affecting the most nodes first.
Insufficient cpu / Insufficient memory
The most common, and the most misunderstood.
The scheduler compares your Pod's requests against each node's unreserved capacity — not against actual usage. A node sitting at 15% real CPU can be completely unschedulable because existing Pods have requested everything.
kubectl describe node ip-10-0-1-42 | grep -A8 "Allocated resources"
Allocated resources:
Resource Requests Limits
cpu 3800m (95%) 7200m (180%)
memory 6.1Gi (81%) 12Gi (160%)
Limits above 100% is normal overcommit. Requests at 95% means the node is full, whatever kubectl top node says.
Three fixes, in order of preference:
Lower the Pod's requests if they exceed what the workload genuinely needs. Over-requesting is extremely common, since requests are usually guessed once and never revisited. Compare against real usage with kubectl top pod before assuming you need more nodes.
Right-size the existing Pods. If every node is 90% requested but 20% used, the cluster is not full — it is over-reserved. This is also where most cluster overspending lives.
Add capacity, via Cluster Autoscaler or Karpenter. Note that a Pod requesting more than any single node can provide will never schedule regardless of how many nodes exist:
kubectl get nodes -o custom-columns=\
NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
Untolerated taint
A taint marks a node as unsuitable unless a Pod explicitly tolerates it. Control-plane nodes carry one by default, which is why they appear in almost every one of these messages and can usually be ignored.
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
If a taint is blocking nodes you want to use — a GPU pool, a dedicated node group — add a toleration:
spec:
tolerations:
- key: "workload"
operator: "Equal"
value: "gpu"
effect: "NoSchedule"
Watch for taints you did not add. node.kubernetes.io/disk-pressure, memory-pressure or not-ready are applied automatically by the kubelet and mean the node has a problem — the answer there is fixing the node, not tolerating it.
Volume node affinity conflict
The Pod uses a PersistentVolumeClaim bound to a volume in one Availability Zone, and no suitable node exists in that zone.
Zonal block storage — EBS, GCP Persistent Disk, Azure Disk — can only attach within its own zone, so the volume pins the Pod there.
kubectl get pv <pv-name> -o jsonpath='{.spec.nodeAffinity}'
kubectl get nodes -L topology.kubernetes.io/zone
The root cause is nearly always a StorageClass using volumeBindingMode: Immediate, which provisions the disk before the scheduler has chosen a node. Switch to WaitForFirstConsumer, which defers provisioning until a node is picked so the disk lands in the right zone:
volumeBindingMode: WaitForFirstConsumer
That fixes future volumes. An existing mismatched one must be migrated via a snapshot.
Pod has unbound immediate PersistentVolumeClaims
Different message, different problem — the PVC itself has not bound.
kubectl get pvc
kubectl describe pvc <name>
If the event says waiting for first consumer to be created before binding, nothing is wrong — that is WaitForFirstConsumer working as designed, and it resolves once the Pod is scheduled. Otherwise the usual causes are no default StorageClass, no CSI driver installed, or an access mode the backing storage cannot provide (commonly ReadWriteMany on block storage).
didn't match Pod's node affinity/selector
A nodeSelector or nodeAffinity rule matches no node. Usually a label typo, or a label that exists on no node at all:
kubectl get nodes --show-labels
kubectl get pod <pod> -o jsonpath='{.spec.nodeSelector}'
didn't match pod anti-affinity rules
Anti-affinity requiring replicas on separate nodes cannot be satisfied because there are fewer eligible nodes than replicas. Three replicas with hard anti-affinity on a two-node cluster leaves one permanently Pending.
Either add nodes, or relax the rule from requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution, which makes it best-effort.
This is also why single-node clusters are misleading for testing — the rule silently does nothing there and then blocks scheduling in production.
Too many pods
1 node(s) had reached the max pod limit
Nodes cap how many Pods they run. On EKS with the VPC CNI this is driven by ENI limits, so a t3.medium tops out at 17 Pods regardless of how idle it looks. Enabling prefix delegation raises it substantially.
kubectl get nodes -o custom-columns=NAME:.metadata.name,MAXPODS:.status.allocatable.pods
A checklist
kubectl describe pod <pod>and read theFailedSchedulingevent in full — every node has a stated reason.Insufficient cpu/memory→ compare requests againstAllocated resources, not actual usage.- Are your requests genuinely needed? Over-requesting is the most common cause.
untolerated taint→ control-plane taints are expected;disk-pressureand similar mean a broken node.volume node affinity conflict→ switch the StorageClass toWaitForFirstConsumer.unbound PersistentVolumeClaims→ check for a default StorageClass and a CSI driver.didn't match node selector→ comparekubectl get nodes --show-labelsagainst the Pod's selector.max pod limit→ ENI limits on EKS; enable prefix delegation.
Frequently Asked Questions
What does "0/5 nodes are available" mean?
The scheduler evaluated all five nodes and rejected every one, and the rest of the message states why for each. The numbers sum to your node count, so 2 Insufficient cpu, 2 untolerated taint, 1 volume node affinity conflict accounts for all five. Treat it as a checklist rather than a single error: fix whichever reason affects the most nodes first, since that is the one standing between you and a schedulable Pod.
Why does Kubernetes say Insufficient cpu when my nodes look idle?
Because scheduling works from the sum of Pod requests on each node, never from measured usage. A node running at 15% actual CPU can be fully committed if the Pods on it requested all its allocatable capacity. Run kubectl describe node <name> and read the "Allocated resources" section, which shows requests as a percentage of capacity. Requests far above real usage are extremely common and are also where most cluster overspending comes from.
What is a volume node affinity conflict?
The Pod needs a PersistentVolumeClaim bound to a volume in one Availability Zone, and the scheduler cannot place the Pod on a node in that zone. Zonal block storage such as EBS can only attach within its own zone, so the volume effectively pins the Pod. The root cause is usually a StorageClass with volumeBindingMode: Immediate, which creates the disk before a node is chosen. Switching to WaitForFirstConsumer prevents it for new volumes; an existing mismatched volume needs migrating via a snapshot.
My PVC says "waiting for first consumer". Is that an error?
No — it is volumeBindingMode: WaitForFirstConsumer behaving exactly as intended. The StorageClass deliberately defers provisioning until the scheduler picks a node, so the disk is created in the right Availability Zone rather than an arbitrary one. It binds as soon as a Pod that uses it is scheduled. Genuine PVC problems produce different messages, typically about no available persistent volumes or no default StorageClass.
Why is one replica always Pending when the others are running?
Usually pod anti-affinity requiring replicas on distinct nodes, with fewer eligible nodes than replicas — three replicas with a hard anti-affinity rule on a two-node cluster leaves one permanently unschedulable. Either add nodes, or change requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution so the rule becomes best-effort. The other candidate is a ReadWriteOnce volume already attached to a different node.
How do I fix "node(s) had reached the max pod limit"?
Nodes have a cap on concurrent Pods, visible via kubectl get nodes -o custom-columns=NAME:.metadata.name,MAXPODS:.status.allocatable.pods. On EKS with the AWS VPC CNI that limit comes from how many IP addresses the instance's network interfaces can carry — a t3.medium allows only 17 Pods regardless of its CPU and memory being idle. Enable prefix delegation on the CNI, which assigns /28 blocks per interface and raises the ceiling substantially, or use larger instance types.