Persistent Volumes and PersistentVolumeClaims

A PersistentVolume (PV) is storage that exists in the cluster. A PersistentVolumeClaim (PVC) is a namespaced request for some of it. A pod mounts the claim, never the volume.

The indirection is the point: an application manifest asks for "20Gi, ReadWriteOnce, fast" and works unchanged whether that resolves to an EBS volume, a Ceph RBD image, or a directory on a laptop.

Scope and ownership

Because a PV is cluster-scoped and a PVC is namespaced, a PVC in staging can never bind a PV already claimed by production. Bindings are exclusive and one-to-one.

Static versus dynamic provisioning

Static means an administrator creates PVs ahead of time and claims bind to whichever one fits. You will see this in tutorials and on bare metal, and rarely anywhere else:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: demo-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /data/demo-pv        # local clusters only, never production

Dynamic means a StorageClass creates the PV on demand when a PVC appears. This is how essentially every real cluster works:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 1Gi

apiVersion: v1
kind: Pod
metadata:
  name: pvc-demo
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ['sh', '-c', 'echo hello > /data/hello.txt && sleep 3600']
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: demo-pvc

Omitting storageClassName entirely uses the cluster's default StorageClass. Setting it to "" explicitly disables dynamic provisioning and forces binding to a pre-created PV. A distinction that catches people out, because the two are not the same thing.

Binding states

kubectl get pv,pvc

NAME                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM
persistentvolume/pvc-a1b2   20Gi       RWO            Delete           Bound      payments/payments-data

NAME                              STATUS   VOLUME      CAPACITY   ACCESS MODES
persistentvolumeclaim/payments-data   Bound    pvc-a1b2    20Gi       RWO

Released is the state worth understanding. A released PV is not reusable as-is: it still carries a claimRef pointing at the deleted PVC, so no new claim can bind to it even though the data is intact. Recovering it means clearing that reference. See below.

Access modes

Two things people get wrong here. RWO is a node constraint, so several pods on the same node can share an RWO volume, which makes the failure look intermittent, working until the scheduler spreads replicas. And block storage cannot do RWX at all: EBS, GCP Persistent Disk and Azure Disk are RWO only, so shared read-write requires EFS, Azure Files, Filestore or NFS.

ReadWriteOncePod is the strict one, useful when even two pods on the same node writing concurrently would corrupt data.

Reclaim policies

Dynamic provisioning defaults to Delete on nearly every cloud StorageClass. That means deleting a namespace destroys its database volumes with it, permanently.

For anything valuable, use a StorageClass with reclaimPolicy: Retain. You can also flip an existing PV:

kubectl patch pv pvc-a1b2 -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

This is a five-second change that has saved a great many people.

Expanding a volume

kubectl patch pvc payments-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Requirements and caveats:

• The StorageClass must have allowVolumeExpansion: true.
• Shrinking is never supported. Only grow.
• The filesystem resize may need a pod restart; check the PVC's conditions for FileSystemResizePending.

kubectl get pvc payments-data -o jsonpath='{.status.conditions}'

Recovering data from a Released PV

The scenario: someone deleted a PVC, the policy was Retain, and the data is still on the volume. The PV is Released and will not bind to anything.

# 1. Confirm the data is still there
kubectl get pv pvc-a1b2 -o yaml

# 2. Remove the stale claim reference so it returns to Available
kubectl patch pv pvc-a1b2 -p '{"spec":{"claimRef": null}}'

# 3. Create a PVC that targets it explicitly

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: payments-data-restored
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: ""          # disable dynamic provisioning
  volumeName: pvc-a1b2          # bind to this specific PV
  resources:
    requests:
      storage: 20Gi

A PVC stuck in Terminating

Almost always a finalizer waiting on something still using the volume.

kubectl get pvc payments-data -o jsonpath='{.metadata.finalizers}'

The kubernetes.io/pvc-protection finalizer holds the PVC while any pod still references it. Find and remove the consumer:

kubectl describe pvc payments-data | grep -A3 "Used By"

Force-removing the finalizer is possible and is a genuinely bad idea unless you are certain nothing is attached. You can end up with a volume attached to a node that Kubernetes no longer tracks.

Storage for StatefulSets

A Deployment sharing one PVC does not scale past one replica on RWO storage. When each replica needs its own volume, that is a StatefulSet with a volumeClaimTemplate:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:17
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: [ReadWriteOnce]
        storageClassName: gp3
        resources:
          requests:
            storage: 20Gi

This creates data-postgres-0, data-postgres-1, data-postgres-2, each bound to its own volume and reattached to the same pod ordinal after rescheduling.

One thing to know: deleting a StatefulSet does not delete its PVCs. That is deliberate protection, and it also means scaling down then up reuses the old data, and that orphaned PVCs keep costing money until someone removes them.

Quick reference

---

Key takeaways

• PVs are cluster-scoped, PVCs are namespaced, and bindings are exclusive one-to-one.
• Dynamic provisioning is the norm: write the PVC, let the StorageClass make the PV.
• RWO is per node, not per pod, so the failure only appears once replicas spread across nodes.
• Block storage cannot do RWX. Shared read-write needs a file-based backend.
• Delete is the usual default reclaim policy. Switch valuable volumes to Retain. It is a one-line patch.
• A Released PV will not rebind until its stale claimRef is cleared.
• StatefulSet PVCs outlive the StatefulSet, protecting data and quietly accruing cost.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between a PV and a PVC?

A PersistentVolume is the storage itself and is cluster-scoped; a PersistentVolumeClaim is a namespaced request for storage with a given size, access mode and class. Pods reference claims, never volumes directly. The separation lets application teams ask for storage in portable terms while the platform decides what actually backs it, so the same manifest works across clusters with entirely different storage. With dynamic provisioning you write only the PVC and a StorageClass creates the matching PV.

Why is my PVC stuck in Pending?

Check kubectl describe pvc <name> first. "Waiting for first consumer" is normal WaitForFirstConsumer behaviour and resolves once a pod uses the claim. Genuine causes are no default StorageClass, no CSI driver installed so nothing can provision, an access mode the backing storage does not support, most commonly ReadWriteMany on block storage, insufficient capacity in a statically provisioned pool, or a zone mismatch between an existing volume and where the pod can schedule.

How do I resize a PersistentVolumeClaim?

Patch the requested storage upward: kubectl patch pvc <name> -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'. This requires allowVolumeExpansion: true on the StorageClass. Shrinking is never supported by any provisioner. Some drivers complete the filesystem resize online while others set a FileSystemResizePending condition and finish when the pod restarts, so check the PVC's status conditions if the new size does not appear inside the container.

Can I recover data after accidentally deleting a PVC?

Only if the reclaim policy was Retain. In that case the PV moves to Released with the data intact but will not bind to anything, because it still references the deleted claim. Clear it with kubectl patch pv <pv> -p '{"spec":{"claimRef": null}}', then create a PVC with volumeName set to that PV and storageClassName: "" so dynamic provisioning does not hand you a new empty volume instead. With Delete, the underlying storage was destroyed and only a snapshot or backup will help.

Why is my PVC stuck Terminating?

The kubernetes.io/pvc-protection finalizer is holding it because a pod still references the claim. Run kubectl describe pvc <name> and look at the "Used By" field, then delete or reschedule that pod. The finalizer clears automatically once nothing is using the volume. Removing the finalizer by hand does work but risks leaving a volume attached to a node that Kubernetes no longer knows about, so it should be a last resort after confirming nothing is mounted.

Do StatefulSet volumes get deleted when I delete the StatefulSet?

No. PVCs created from a volumeClaimTemplate deliberately outlive the StatefulSet so that deleting a workload cannot destroy a database. Recreating the StatefulSet with the same name reattaches the existing volumes, which is also how scaling down and back up preserves data. The flip side is that finished workloads leave PVCs behind that continue to cost money, so decommissioning something properly means deleting its claims explicitly.