Unable to attach or mount volumes: timed out waiting for the condition

The kubelet asked the CSI driver to attach a volume and gave up waiting. The Pod sits in ContainerCreating indefinitely. There are four distinct causes and the event text tells you which.

kubectl describe pod my-pod

Multi-Attach: the most common one

ReadWriteOnce means the volume can be mounted read-write by one node at a time. Not one Pod: one node. This trips people up during rolling updates.

Multi-Attach error for volume "pvc-9f2a..."
Volume is already used by pod(s) api-7d4f8c9b6-xk2mn

A Deployment with strategy: RollingUpdate starts the new Pod before terminating the old one. If they land on different nodes, the new one cannot attach and waits for the old one to release, which only happens once the new one is Ready. Deadlock.

Find the holder:

kubectl get pods -o wide --all-namespaces \
  | grep -E "api-7d4f8c9b6-xk2mn"

For a single-replica workload with a ReadWriteOnce volume, use Recreate:

spec:
  strategy:
    type: Recreate

The old Pod terminates fully before the new one starts. You accept a few seconds of downtime, which for a workload that cannot run two copies anyway is not a real loss.

A StatefulSet avoids this by design: each replica gets its own PVC from volumeClaimTemplates, and the controller never runs two Pods with the same ordinal.

If you genuinely need several Pods writing the same volume, you need ReadWriteMany, which means a file-based backend such as EFS, Azure Files or an NFS server. EBS and GCE persistent disks are block devices and cannot do it.

Zone mismatch

A zonal disk exists in one availability zone. A node in another zone cannot attach it, ever.

kubectl get pv pvc-9f2a... -o jsonpath='{.spec.nodeAffinity}' | jq
kubectl get node ip-10-0-2-15 -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}'

Disk in eu-west-1a, node in eu-west-1b, and the Pod waits forever.

The prevention is volumeBindingMode: WaitForFirstConsumer on the StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3

With Immediate, the volume is created as soon as the PVC exists, in whichever zone the provisioner picks, and the scheduler must then find a node there. With WaitForFirstConsumer, the scheduler places the Pod first and the volume is created in that Pod's zone. It is the correct setting for zonal storage in a multi-zone cluster, and the default on most managed clusters now.

Changing it does not move existing volumes. A PVC already bound to a disk in the wrong zone has to be recreated, which means restoring from a snapshot if the data matters.

Stale attachment after a node failure

When a node dies hard, the controller may never get to detach the volume, leaving a VolumeAttachment pointing at a node that no longer exists.

kubectl get volumeattachment \
  -o custom-columns=NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached

NAME       PV           NODE           ATTACHED
csi-8f2a   pvc-9f2a     ip-10-0-1-99   true

If ip-10-0-1-99 is gone from kubectl get nodes, that attachment is stale and blocks every future attach:

kubectl delete volumeattachment csi-8f2a

Check the cloud console too. AWS occasionally leaves an EBS volume in attaching or attached to a terminated instance, and only a detach there clears it:

aws ec2 describe-volumes --volume-ids vol-0abc123 \
  --query 'Volumes[0].Attachments'
aws ec2 detach-volume --volume-id vol-0abc123 --force

--force can corrupt a filesystem that is still being written to. Only use it once you are certain the instance is gone.

The CSI driver itself

If several volumes are stuck at once, suspect the driver rather than any one volume:

kubectl -n kube-system get pods -l app=ebs-csi-controller
kubectl -n kube-system logs -l app=ebs-csi-controller -c ebs-plugin --tail=50
kubectl -n kube-system get pods -l app=ebs-csi-node -o wide

The node plugin is a DaemonSet, so a node missing its CSI node Pod cannot mount anything. That shows up as a single node where everything with a volume fails.

IAM is the other frequent cause on EKS. The controller needs ec2:AttachVolume, ec2:DetachVolume, ec2:CreateVolume and ec2:DescribeVolumes; a missing permission surfaces as an rpc error in the Pod events with the real reason in the controller log.

Instance attachment limits

Every instance type has a maximum number of attached volumes, and Nitro instances count ENIs and the root volume towards it. Once a node is at its limit, further attaches fail even though the disk and zone are fine.

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=ip-10-0-1-42 | wc -l

The scheduler is aware of this via the CSI driver's reported limit, so it usually avoids overfilling a node. It gets it wrong when volumes are attached outside Kubernetes.

A checklist

1. kubectl describe pod and read the exact event.
2. Multi-Attach → find the holding Pod. Single replica → strategy: Recreate.
3. Need several writers → you need ReadWriteMany, so EFS or Azure Files, not EBS.
4. Compare the PV's zone against the node's topology.kubernetes.io/zone.
5. Set volumeBindingMode: WaitForFirstConsumer to stop it recurring.
6. kubectl get volumeattachment and delete any pointing at a node that is gone.
7. Several volumes stuck → check the CSI controller and node DaemonSet logs.
8. On EKS, confirm the controller's IAM role has the EC2 volume permissions.

---

Frequently Asked Questions

What does "Multi-Attach error for volume" mean?

A ReadWriteOnce volume can be mounted read-write by only one node at a time, and a second node is trying to attach it. The usual trigger is a rolling update where the new Pod is scheduled to a different node than the old one: the new Pod cannot attach until the old one releases, and the old one is not terminated until the new one is Ready, so the two deadlock. For a single-replica workload the fix is strategy: Recreate, which terminates the old Pod first.

How do I fix a zone mismatch between a PersistentVolume and a node?

You cannot move a zonal disk, so the existing PVC must be recreated in the right zone, restoring from a snapshot if the data matters. To stop it recurring, set volumeBindingMode: WaitForFirstConsumer on the StorageClass. That defers volume creation until the scheduler has chosen a node, so the disk is created in the zone where the Pod actually is. With the older Immediate mode the disk is created first, in whichever zone the provisioner picks, and the scheduler is then constrained to that zone.

Why is my volume still attached to a node that no longer exists?

When a node terminates abruptly the controller may never complete the detach, leaving a VolumeAttachment object referencing the dead node. That object blocks any future attach of the volume. List them with kubectl get volumeattachment and delete the stale one. Check the cloud provider too: AWS sometimes leaves an EBS volume attached to a terminated instance, which needs aws ec2 detach-volume, and the --force flag there can corrupt a filesystem if the instance is somehow still writing.

Can several Pods share one PersistentVolumeClaim?

Only if the access mode is ReadWriteMany, and only with a storage backend that supports it. Block storage such as EBS, GCE persistent disks and Azure managed disks cannot, because a block device mounted read-write by two hosts corrupts the filesystem. File-based backends can: EFS, Azure Files, and NFS. Note that ReadWriteOnce permits multiple Pods on the same node to share the volume, which is why it sometimes appears to work until the scheduler places a replica elsewhere.

Why does a Pod hang in ContainerCreating with no error at all?

Check whether the PVC is simply waiting: with volumeBindingMode: WaitForFirstConsumer, a PVC stays Pending with a waiting for first consumer message until a Pod that uses it is scheduled, which is normal rather than a fault. If the PVC is Bound and the Pod still hangs, look at the CSI node plugin on that specific node, since the mount happens there. A node missing its CSI DaemonSet Pod fails every volume mount while the rest of the cluster works fine.