Kubernetes Storage

Pods are designed to be thrown away. Anything written to a container's own filesystem dies with it, which is exactly right for a stateless API and exactly wrong for a database.

Kubernetes storage is the set of abstractions that let data outlive the pod using it, while keeping application teams out of the business of provisioning disks.

Everything a container writes is temporary by default

A container's writable layer lives for the life of that container. Restart it, not the pod, just the container after a crash, and the layer is recreated empty. Nothing you wrote survives.

That surprises people who see their pod still Running with a restart count of 1 and wonder where their files went.

The storage options, in order of how long data lasts

emptyDir

Created empty when the pod is assigned to a node, deleted when the pod is removed. Its main use is sharing a directory between containers in the same pod:

spec:
  volumes:
    - name: scratch
      emptyDir: {}
  containers:
    - name: app
      volumeMounts:
        - name: scratch
          mountPath: /var/cache/app
    - name: log-shipper
      volumeMounts:
        - name: scratch
          mountPath: /logs
          readOnly: true

It consumes node disk, and by default nothing bounds it. A runaway process filling an emptyDir can fill the node and get every pod on it evicted. Set a limit:

emptyDir:
  sizeLimit: 1Gi

hostPath: almost always the wrong answer

hostPath mounts a directory from the node itself. It survives pod deletion, but only on that node, so the moment the pod is rescheduled elsewhere the data is gone. It is also a serious security concern: a pod that can mount / or the container runtime socket effectively owns the node.

Legitimate uses are node-level agents. Log collectors reading /var/log, monitoring agents reading /proc. For application data, use a PersistentVolumeClaim.

PersistentVolumes and PersistentVolumeClaims

The abstraction has two halves, and the split exists to separate two audiences.

• A PersistentVolume (PV) is a piece of storage in the cluster. Cluster-scoped. Usually created automatically.
• A PersistentVolumeClaim (PVC) is a request for storage from inside a namespace: this much, with these access modes, of this class.

An application team writes a PVC. The platform provides the PV. Developers never name a disk ID or a NFS export, so the same manifest works on a laptop, in staging, and on a cloud cluster where the backing storage is completely different.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: payments-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gp3
  resources:
    requests:
      storage: 20Gi

# Consuming it from a pod
spec:
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: payments-data
  containers:
    - name: db
      volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data

In practice you rarely write a PV by hand. A StorageClass provisions one dynamically when the PVC is created, that is what storageClassName: gp3 above is asking for.

Access modes

This is where expectations most often break.

The critical detail: RWO is per node, not per pod. Two pods on the same node can both mount an RWO volume. Two pods on different nodes cannot, which is why a Deployment with 3 replicas and one RWO PVC leaves two pods stuck in Pending. The volume is already attached elsewhere.

And most block storage is RWO only. AWS EBS, GCP Persistent Disk and Azure Disk cannot do RWX at all. Shared read-write access needs a file system: EFS, Azure Files, Filestore, or NFS.

If you need many replicas each with their own storage, that is a StatefulSet with a volumeClaimTemplate, not a Deployment sharing one PVC.

Reclaim policy: what happens when the claim goes

Delete is the default on most cloud StorageClasses, which means deleting a namespace can permanently destroy a database volume. For anything you would be upset to lose, use a StorageClass with reclaimPolicy: Retain, and rely on volume snapshots rather than hoping.

CSI: where the implementations come from

Kubernetes itself does not know how to talk to EBS or Ceph. The Container Storage Interface is the plugin standard, and drivers are installed separately. The EBS CSI driver on EKS, and so on. This matters when a PVC stays Pending and the events say no provisioner is available: the driver is not installed.

CSI drivers also supply volume snapshots and volume expansion, which is how you grow a PVC:

kubectl patch pvc payments-data -p '{"spec":{"resources":{"requests":{"storage":"50Gi"}}}}'

Expansion works only if the StorageClass sets allowVolumeExpansion: true, and shrinking is never supported.

Debugging a Pending PVC

kubectl get pvc
kubectl describe pvc payments-data

Events:
  Type     Reason              Message
  ----     ------              -------
  Normal   WaitForFirstConsumer  waiting for first consumer to be created before binding

That particular message is not an error. It is volumeBindingMode: WaitForFirstConsumer working correctly, deferring provisioning until the scheduler picks a node so the disk is created in the right Availability Zone. It binds as soon as a pod uses it.

Real causes of a stuck PVC:

• No StorageClass and no default: kubectl get storageclass should show one marked (default).
• No CSI driver installed, so nothing provisions the volume.
• Access mode unsupported by the backing storage, typically RWX on block storage.
• Zone mismatch: the volume is in one Availability Zone and the pod cannot be scheduled there.

Explore further in this section

Quick reference

---

Key takeaways

• The container filesystem does not survive a container restart. Not just pod deletion. A crash restart loses it too.
• emptyDir dies with the pod, and without sizeLimit it can fill the node and trigger evictions.
• hostPath is node-local and a security risk. Use it for node agents, never for application data.
• RWO means one node, not one pod. It is why a multi-replica Deployment on one PVC leaves pods Pending.
• Block storage cannot do RWX. Shared read-write needs EFS, Azure Files or NFS.
• Delete is the common default reclaim policy: deleting a namespace can destroy a database volume.
• WaitForFirstConsumer is not a bug; it defers binding so the disk lands in the right zone.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between a PersistentVolume and a PersistentVolumeClaim?

A PersistentVolume is the actual storage resource in the cluster and is cluster-scoped. A PersistentVolumeClaim is a namespaced request for storage. This much capacity, these access modes, this StorageClass. The split separates the platform team who supply storage from the application team who consume it, so a manifest asking for "20Gi ReadWriteOnce" works unchanged whether the backing store is EBS, Ceph or a local disk. With dynamic provisioning you write only the PVC and the StorageClass creates the PV automatically.

Why is my PersistentVolumeClaim stuck in Pending?

Run kubectl describe pvc <name> and read the events. If it says "waiting for first consumer", nothing is wrong. The StorageClass uses WaitForFirstConsumer and binds once a pod schedules. Otherwise the usual causes are no default StorageClass (kubectl get storageclass should show one marked default), no CSI driver installed so nothing can provision, an access mode the backing storage cannot provide such as ReadWriteMany on block storage, or a zone mismatch between an existing volume and where the pod can be scheduled.

Can multiple pods share the same PersistentVolume?

It depends entirely on the access mode and the backing storage. ReadWriteOnce allows read-write mounting by one node at a time, so pods on that same node can share it but pods elsewhere cannot. ReadWriteMany allows many nodes simultaneously, but block storage, EBS, GCP Persistent Disk, Azure Disk, cannot provide it at all. For genuine shared read-write access you need a file-based system such as EFS, Azure Files, Filestore or NFS. If each replica needs its own storage rather than shared storage, use a StatefulSet with a volumeClaimTemplate.

What happens to my data when I delete a PVC?

That is set by the reclaim policy on the PersistentVolume. With Delete, which is the default for most cloud StorageClasses, the underlying disk is destroyed along with everything on it. With Retain, the PV and data survive and an administrator decides what to do next. Because Delete is the default, removing a namespace can permanently destroy a database volume. For anything valuable, use a StorageClass with reclaimPolicy: Retain and take volume snapshots.

Should I ever use hostPath?

Only for workloads that genuinely need access to the node's own filesystem. Log collectors reading /var/log, monitoring agents reading /proc, some CSI and networking components. For application data it is a poor choice: the data is bound to one node, so a rescheduled pod finds nothing, and it is a significant security risk since a pod able to mount host paths can often escalate to controlling the node. Use a PersistentVolumeClaim instead, or emptyDir if the data is genuinely temporary.

What is emptyDir and when should I use it?

emptyDir is a directory created when a pod is assigned to a node and deleted when the pod is removed. It survives container restarts but not pod replacement, which makes it right for scratch space, caches, and sharing files between containers in the same pod. An application writing logs that a sidecar ships, for instance. Set a sizeLimit, because by default it can consume all node disk and cause evictions across the node. Setting medium: Memory makes it a tmpfs, which is fast but counts against the pod's memory limit.