PersistentVolumeClaim stuck in Pending

A Pending PVC means Kubernetes has nothing to bind the claim to, and the Pod waits indefinitely as a result, 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims is the scheduler reporting that it cannot place a Pod whose storage does not exist.

Four causes account for nearly all of it, and the PVC's own events distinguish them.

Read the events first

kubectl describe pvc data-pvc | tail -8

Events:
  Type     Reason              Age   From                         Message
  Warning  ProvisioningFailed  2m    persistentvolume-controller  storageclass.storage.k8s.io "gp3" not found

kubectl get sc
kubectl get pvc data-pvc -o jsonpath='{.spec.storageClassName}{"\n"}'

The message maps directly to the cause:

Cause 1: No CSI driver installed

The most common cause on a new cluster, and the one with the least helpful symptom: the PVC sits Pending with no events whatsoever, because nothing exists to act on it.

kubectl get pods -n kube-system | grep -i csi

(no output)

Since Kubernetes 1.23 the in-tree cloud provisioners have been removed, so a cluster needs the CSI driver for its platform installed separately. On EKS that is an add-on a fresh cluster does not include:

aws eks create-addon --cluster-name prod \
  --addon-name aws-ebs-csi-driver \
  --service-account-role-arn "$EBS_CSI_ROLE_ARN"

The role matters. The driver needs EC2 permissions to create volumes, and without IRSA configured it installs, runs, and fails every provisioning attempt with an access denied in its logs:

kubectl logs -n kube-system -l app=ebs-csi-controller -c ebs-plugin --tail=30

Pending PVCs bind on their own once the driver is working. No need to recreate them.

Cause 2: The storage class does not exist, or there is no default

storageclass.storage.k8s.io "gp3" not found

kubectl get sc

NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      DEFAULT
gp2 (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true

The manifest asks for gp3 and the cluster has gp2. Either create the class or change the claim.

Creating gp3 is usually worth doing anyway, since it is cheaper and faster than gp2:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Delete
parameters:
  type: gp3
  encrypted: "true"

The other variant is a claim with no storageClassName on a cluster with no default class:

no persistent volumes available for this claim and no storage class is set

kubectl patch storageclass gp3 \
  -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Make sure only one class is marked default. Two produces unpredictable binding.

Cause 3: "Waiting for first consumer" is not a problem

Normal  WaitForFirstConsumer  persistentvolume-controller
waiting for first consumer to be created before binding

This is correct behaviour, not a fault. A storage class with volumeBindingMode: WaitForFirstConsumer deliberately delays creating the volume until a Pod is scheduled, so the volume is created in the same availability zone as the node that will use it.

If the PVC stays here, the problem is the Pod, not the claim:

kubectl describe pod api-7d4f-x8k2 | tail -8

Usually the Pod is unschedulable for an unrelated reason, insufficient CPU, a taint, a node selector, and fixing that binds the PVC automatically.

The alternative mode, Immediate, creates the volume straight away and is what causes cause 4.

Cause 4: Zone mismatch

Warning  FailedScheduling  default-scheduler
0/3 nodes are available: 3 node(s) had volume node affinity conflict

An EBS volume exists in one availability zone and can only attach to a node in that zone. With volumeBindingMode: Immediate, the volume is created before anything knows where the Pod will run, so it can land in a zone with no capacity, or no nodes at all.

kubectl get pv -o custom-columns=NAME:.metadata.name,ZONE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]
kubectl get nodes -L topology.kubernetes.io/zone

NAME       ZONE
pvc-a3f2   eu-west-1c

NAME              ZONE
ip-10-0-1-42      eu-west-1a
ip-10-0-2-17      eu-west-1b

The volume is in a zone with no nodes.

The fix is WaitForFirstConsumer on the storage class, which prevents the whole class of problem. For the stranded volume, either add a node in that zone or delete the PVC and let it be recreated once the binding mode is corrected. After taking a snapshot if it holds anything.

Cause 5: Unsupported access mode

kubectl get pvc data-pvc -o jsonpath='{.spec.accessModes}{"\n"}'

["ReadWriteMany"]

EBS does not support ReadWriteMany. A block device attaches to one node at a time, so a claim asking several Pods to mount it read-write cannot be satisfied and simply stays Pending.

If you genuinely need shared access across Pods, use EFS with the EFS CSI driver. If you do not, and often a StatefulSet giving each replica its own volume is the right shape. Change the mode to ReadWriteOnce.

Note ReadWriteOnce means one node, so several Pods on the same node can share it. ReadWriteOncePod restricts it to a single Pod, which is what you want for a database.

Checking a bound volume did what you expected

kubectl get pvc,pv

NAME             STATUS   VOLUME     CAPACITY   ACCESS MODES   STORAGECLASS
pvc/data-pvc     Bound    pvc-a3f2   20Gi       RWO            gp3

NAME         CAPACITY   RECLAIM POLICY   STATUS   CLAIM
pv/pvc-a3f2  20Gi       Delete           Bound    default/data-pvc

Worth noticing RECLAIM POLICY: Delete. Deleting the PVC destroys the volume and its data. For anything you care about, use a class with reclaimPolicy: Retain, or accept that kubectl delete pvc is a destructive command.

A checklist

1. kubectl describe pvc <name>. The events name the cause.
2. No events at all → no CSI driver. kubectl get pods -n kube-system | grep csi.
3. storageclass not found → kubectl get sc and compare with the claim.
4. no storage class is set → mark one class default.
5. WaitForFirstConsumer → normal; debug the Pod instead.
6. volume node affinity conflict → zone mismatch; switch to WaitForFirstConsumer.
7. ReadWriteMany on EBS → impossible; use EFS or change the mode.
8. Driver logs: kubectl logs -n kube-system -l app=ebs-csi-controller -c ebs-plugin.

---

Frequently Asked Questions

Why is my PersistentVolumeClaim stuck in Pending?

Kubernetes has nothing to bind it to, and kubectl describe pvc names the reason in its events. The four common causes are: no CSI driver installed, so nothing is listening for claims at all; a storage class that does not exist or no default class set; a zone mismatch where the volume and the nodes are in different availability zones; and an access mode the backend cannot provide. A PVC with no events whatsoever almost always means the driver is missing.

Why does a new EKS cluster fail to provision volumes?

Because the EBS CSI driver is not included. Since Kubernetes 1.23 the in-tree cloud provisioners were removed, so volume provisioning requires the CSI driver installed as a separate add-on, and a freshly created EKS cluster does not have it. Install aws-ebs-csi-driver with an IRSA role granting the EC2 permissions it needs; without the role it runs but fails every provisioning attempt with access denied in its logs. Pending claims bind by themselves afterwards.

What does "waiting for first consumer to be created" mean?

That the storage class uses volumeBindingMode: WaitForFirstConsumer, which deliberately delays creating the volume until a Pod using the claim is scheduled, so the volume can be created in the same availability zone as the node. It is correct behaviour rather than an error. If the PVC stays in that state, the problem is that the Pod cannot be scheduled: check the Pod's events for insufficient resources, taints or node selectors.

What causes "volume node affinity conflict"?

The volume exists in one availability zone and no schedulable node is in that zone, so the Pod cannot be placed anywhere that can attach it. EBS volumes are zonal, and with volumeBindingMode: Immediate the volume is created before the scheduler has decided where the Pod will run, so it can easily land in the wrong zone. Using WaitForFirstConsumer prevents this entirely by creating the volume only once the node is known.

Why can't I use ReadWriteMany with EBS?

Because EBS is block storage and a volume attaches to one node at a time, so several Pods on different nodes cannot mount it read-write simultaneously. The claim is unsatisfiable and stays Pending with no useful error. For genuinely shared access, use EFS with the EFS CSI driver, which supports ReadWriteMany. Often, though, the better answer is a StatefulSet giving each replica its own ReadWriteOnce volume.

Will deleting a PVC delete my data?

With the default reclaim policy, yes. Most dynamic storage classes use reclaimPolicy: Delete, so removing the PVC destroys the underlying volume and everything on it. Check with kubectl get pv. The RECLAIM POLICY column shows which applies. For anything you care about, use a storage class with reclaimPolicy: Retain, which leaves the volume in place for manual recovery, and take snapshots regardless.