Kubernetes Secrets

A Secret holds sensitive data, passwords, API tokens, TLS keys, registry credentials, as a distinct object type with its own RBAC surface, kept out of describe output and optionally encrypted at rest in etcd.

Start with what it is not, because the misunderstanding here causes real incidents.

Base64 is not encryption

Secret values are base64-encoded in the API. That is an encoding, chosen so binary data survives JSON, and it is reversible by anyone in one command:

kubectl get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d

supersecret

So a Secret gives you no protection at all against a principal who can read Secrets in that namespace. What it genuinely provides:

RBAC is the control that matters. A role granting get on secrets in a namespace grants every secret in it, so audit those grants carefully:

kubectl auth can-i get secrets --as=system:serviceaccount:default:my-app

Encryption at rest must be turned on explicitly via an EncryptionConfiguration on the API server. On managed clusters this usually means enabling envelope encryption with a cloud KMS key, on EKS, for example, it is a cluster setting, not a default.

Creating Secrets

Generic (Opaque)

kubectl create secret generic app-secret \
  --from-literal=DB_PASSWORD=supersecret \
  --from-literal=API_TOKEN=abc123

Note this puts the value in your shell history. --from-file avoids that:

kubectl create secret generic app-secret --from-file=DB_PASSWORD=./db-password.txt

In YAML, use stringData rather than data so you write plaintext and let Kubernetes encode it:

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
type: Opaque
stringData:
  DB_PASSWORD: supersecret     # plaintext in, base64 stored

stringData is write-only. Read the object back and you will see data with base64 values.

Do not commit this file to Git with a real password in it. Base64 is not obfuscation and Git history is permanent. See the production section below.

Registry credentials

kubectl create secret docker-registry regcred \
  --docker-server=ghcr.io \
  --docker-username=acme-ci \
  --docker-password="$GITHUB_TOKEN"

Reference it from the pod:

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: ghcr.io/acme/payments-api:1.4.2

A pod in ImagePullBackOff against a private registry is usually a missing imagePullSecrets, or a Secret in a different namespace, Secrets are namespaced and are not shared across namespaces.

To avoid repeating it in every pod spec, attach it to the namespace's ServiceAccount instead:

kubectl patch serviceaccount default \
  -p '{"imagePullSecrets":[{"name":"regcred"}]}'

TLS

kubectl create secret tls api-example-com-tls \
  --cert=tls.crt --key=tls.key

This creates a kubernetes.io/tls Secret with the fixed keys tls.crt and tls.key, which is the form Ingress expects. In practice let cert-manager create and renew these rather than doing it by hand.

Consuming Secrets

As environment variables

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: app-secret
        key: DB_PASSWORD

Simple, and the most common approach, but environment variables leak more readily than files. They appear in /proc/<pid>/environ, are often captured by crash reporters and APM agents, and are inherited by every child process. For genuinely high-value credentials, prefer a file.

They also never update: changing the Secret does nothing to a running pod until it is replaced.

As mounted files

spec:
  volumes:
    - name: secret-volume
      secret:
        secretName: app-secret
        defaultMode: 0400
  containers:
    - name: app
      volumeMounts:
        - name: secret-volume
          mountPath: /secrets
          readOnly: true

Each key becomes a file under /secrets. These volumes are backed by tmpfs, so the value stays in memory and never lands on node disk, and they do refresh when the Secret changes, within about a minute, and not at all if mounted with subPath.

defaultMode: 0400 is worth setting. Some tools refuse to use a key file that is group- or world-readable; SSH is the classic example.

Production: keep the source of truth outside the cluster

Writing Secret manifests by hand does not scale and cannot be safely committed. The established options:

• External Secrets Operator: syncs from AWS Secrets Manager, SSM Parameter Store, Vault or Azure Key Vault into native Kubernetes Secrets. Git holds only a reference. The most widely used approach.
• Sealed Secrets: encrypts a Secret with a cluster-held public key so the sealed file is safe to commit; only the in-cluster controller can decrypt it. The simplest fit for GitOps.
• Secrets Store CSI Driver: mounts values straight from an external store as files, optionally syncing to a Secret.
• Vault Agent Injector: a sidecar that fetches secrets, with support for dynamic short-lived credentials.

A ExternalSecret looks like this, no secret material in the repository at all:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-secret
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: app-secret
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: prod/payments/db
        property: password

Rotation then happens in one place, and the operator propagates it.

Practical hygiene

• Restrict get secrets by RBAC and review who holds it. It is the only control that stops a read.
• Enable encryption at rest and confirm it is actually on, rather than assuming.
• Don't log secrets. Avoid kubectl describe-style dumps in CI output and be careful with debug logging of config objects.
• Prefer files over environment variables for high-value credentials.
• Set automountServiceAccountToken: false on pods that never call the Kubernetes API. Otherwise every pod carries a token that can be used against the API server.

spec:
  automountServiceAccountToken: false

Quick reference

---

Key takeaways

• Base64 is encoding, not encryption. Anyone with get secrets reads the value in one command.
• RBAC is the real control, plus etcd encryption at rest, which you must enable deliberately.
• Use stringData in YAML so you write plaintext, and never commit that file with a real value.
• Mounted Secrets are tmpfs-backed and refresh; environment variables do neither.
• Secrets are namespaced. A missing imagePullSecrets, or one in the wrong namespace, is the usual private-registry ImagePullBackOff.
• In production, keep the source of truth outside the cluster and sync with the External Secrets Operator or Sealed Secrets.
• Set automountServiceAccountToken: false on pods that never talk to the API.

---

Check your understanding

---

Frequently Asked Questions

Are Kubernetes Secrets encrypted?

Not by default in the way the name implies. Values are base64-encoded in the API, which is trivially reversible with base64 -d. Kubernetes can encrypt Secrets at rest in etcd through an EncryptionConfiguration, and managed clusters generally expose this as envelope encryption with a cloud KMS key, but it must be enabled, and it protects the datastore on disk rather than API reads. An authorised caller always sees plaintext. RBAC is the control that actually restricts access.

How do I decode a Kubernetes Secret?

kubectl get secret <name> -o jsonpath='{.data.<key>}' | base64 -d. The fact that this is a one-liner is the point: base64 is an encoding for safe transport of binary data in JSON, not a security measure. kubectl describe secret deliberately shows key names and byte counts without values, which is safer when sharing terminal output.

What is the difference between data and stringData?

data expects base64-encoded values, so you must encode them yourself. stringData accepts plaintext and Kubernetes encodes it on write, which makes manifests far more readable and removes a whole class of "I forgot to encode it" errors. stringData is write-only. Reading the object back always shows data with base64 values. If both specify the same key, stringData wins.

Why does my pod fail with ImagePullBackOff on a private registry?

Most often the imagePullSecrets reference is missing from the pod spec, or the Secret exists in a different namespace. Secrets are namespaced and are never shared across them. Confirm the Secret is in the pod's namespace and was created with kubectl create secret docker-registry, which produces the required kubernetes.io/dockerconfigjson type. kubectl describe pod quotes the registry's own error, which distinguishes an authentication failure from a genuinely missing tag. To avoid repeating the reference everywhere, attach the Secret to the namespace's ServiceAccount instead.

Should I use environment variables or files for secrets?

Files, for anything high-value. Environment variables are visible in /proc/<pid>/environ, are frequently captured by crash reporters and APM agents, are inherited by child processes, and never refresh when the Secret changes. Mounted Secret volumes are backed by tmpfs so the value never reaches node disk, they update within about a minute when the Secret changes, and you can restrict permissions with defaultMode: 0400. Environment variables remain reasonable for low-sensitivity values where simplicity matters.

How should secrets be managed in production?

Keep the source of truth outside the cluster and sync it in. The External Secrets Operator reads from AWS Secrets Manager, SSM Parameter Store, Vault or Azure Key Vault and creates native Secrets, so Git holds only a reference. Sealed Secrets takes the other approach, encrypting a Secret with a cluster-held key so the encrypted file is safe to commit. The usual fit for GitOps. Either way, no plaintext credential is committed and rotation happens in one place instead of across every manifest.