Kubernetes Probes

Probes are how the kubelet learns whether your container is alive and whether it should receive traffic. Without them, Kubernetes assumes a running process is a working process, which is very often wrong.

They are also one of the easiest things to misconfigure into an outage, so the defaults and the timing deserve real attention.

Three probes, three different questions

The distinction people most often blur is liveness versus readiness. Liveness restarts. Readiness redirects. Failing readiness is a normal, recoverable state. A Pod warming a cache or briefly overloaded should fail readiness and get no traffic, then come back. Failing liveness is a declaration that the process is unrecoverable and must be killed.

Readiness: the one you should always set

Readiness gates Service endpoints. A Pod that is not ready receives no traffic.

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

This is also what makes a rolling update safe. During a Deployment rollout, Kubernetes waits for a new Pod to become Ready before terminating an old one. Without a readiness probe, "running" is taken to mean "ready", so traffic is routed to a process still loading configuration, and the next old Pod is terminated. The rollout reports success while users see errors.

If you set only one probe, set this one.

Liveness: use it sparingly

Liveness exists for the case a process is running but permanently stuck: a deadlock, an exhausted thread pool, an event loop that will never recover.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 20
  timeoutSeconds: 3
  failureThreshold: 3

Ask honestly whether a restart would fix the condition you are detecting. If the answer is no, a liveness probe makes things worse. It adds restarts to a system that is already struggling.

Never check dependencies in a liveness probe. A /healthz that queries the database will fail on every replica the moment the database slows, and Kubernetes will restart your entire fleet in a loop. A slow database becomes a total outage, caused by the health check.

The safe shape: liveness checks only that this process can still respond. Readiness may check whether it can usefully serve.

Startup: for slow boots

Applications that take a long time to start create a dilemma: initialDelaySeconds large enough for a cold JVM makes liveness slow to detect real failures afterwards.

A startup probe resolves it. While it runs, liveness and readiness are suspended entirely:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 30      # allows up to 300s to start
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10
  failureThreshold: 3       # then react in ~30s

That gives five minutes to boot and a thirty-second detection window afterwards. Without it you would need initialDelaySeconds: 300 on liveness, and a container that hung after minute six would go unnoticed for five minutes.

Probe methods

# HTTP, with a header if needed
httpGet:
  path: /healthz
  port: 8080
  httpHeaders:
    - name: X-Probe
      value: kubelet

# TCP, only proves something is listening
tcpSocket:
  port: 5432

# exec. Note this runs a process every period
exec:
  command: ["sh", "-c", "pg_isready -U postgres"]

exec probes are the expensive option. Every probe forks a process inside the container, and an aggressive exec probe across many Pods is a measurable CPU cost. Prefer httpGet where you can.

tcpSocket only proves the port is open, not that the application behind it works. It is a weak check, and for an HTTP service it is usually the wrong one.

The timing fields

Time to detection is periodSeconds × failureThreshold, plus up to one timeoutSeconds. The defaults above mean roughly 30 seconds.

timeoutSeconds: 1 is the default and it is aggressive. An endpoint that occasionally takes 1.2 seconds under load will fail intermittently, and because probes get slower exactly when the system is loaded, the probe pushes a busy service over the edge. Setting it to 3 to 5 seconds costs nothing.

A complete example

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
spec:
  replicas: 3
  selector:
    matchLabels: { app: payments-api }
  template:
    metadata:
      labels: { app: payments-api }
    spec:
      containers:
        - name: api
          image: ghcr.io/acme/payments-api:1.4.2
          ports:
            - containerPort: 8080

          # Allow up to 150s to boot
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 5
            failureThreshold: 30

          # Then detect a wedged process in ~30s
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            periodSeconds: 10
            timeoutSeconds: 3
            failureThreshold: 3

          # Gate traffic; react faster, and allow recovery
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 2

Two endpoints, deliberately different. /healthz answers "this process is alive" and touches nothing external. /ready answers "I can serve a request right now" and may check that a connection pool is initialised, but still not that the whole database is healthy.

Failure modes to recognise

Liveness too aggressive. A slow startup plus a short initialDelaySeconds means the probe fails before the app is up, the container is killed, and it restarts into the same trap. Symptom: CrashLoopBackOff with clean application logs and Liveness probe failed in the events. Use a startup probe.

No readiness probe. Every deploy produces a short burst of errors while Kubernetes routes to Pods that are not serving yet.

Liveness that checks dependencies. A shared dependency degrades and every replica restarts simultaneously. A partial problem becomes a complete outage.

Readiness that never recovers. successThreshold must be 1 for liveness and startup probes, but for readiness a higher value adds hysteresis to a flapping service.

# What the kubelet actually saw
kubectl describe pod <name> | grep -A5 Events

Warning  Unhealthy  Liveness probe failed: Get "http://10.244.1.17:8080/healthz":
                    context deadline exceeded (Client.Timeout exceeded while awaiting headers)

# Is the endpoint even right? Test from inside the Pod.
kubectl exec -it <name> -- wget -qO- localhost:8080/healthz

That last command resolves most probe arguments immediately. Wrong port, wrong path, or an endpoint that requires authentication and returns 401.

Quick reference

---

Key takeaways

• Readiness redirects traffic; liveness restarts the container. Failing readiness is normal, failing liveness is terminal.
• Always set a readiness probe. Without one, rolling updates route traffic to Pods that cannot serve.
• Never check dependencies in a liveness probe: it turns a degraded dependency into a fleet-wide restart loop.
• Use a startup probe for slow boots instead of a large initialDelaySeconds.
• timeoutSeconds defaults to 1, which is too aggressive for most real endpoints.
• Detection takes periodSeconds × failureThreshold. Choose it, do not inherit it.
• exec probes fork a process every period. Prefer httpGet.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between a liveness and a readiness probe?

A liveness probe answers "is this container stuck?" and a failure causes the kubelet to kill and restart it. A readiness probe answers "should this container receive traffic right now?" and a failure removes the Pod from its Service endpoints without restarting anything. Failing readiness is a normal, temporary state. A Pod warming up or briefly overloaded. Failing liveness is a statement that the process is unrecoverable. Use readiness liberally and liveness sparingly.

Why is my Pod in CrashLoopBackOff with "Liveness probe failed"?

The probe is failing before the application is ready to answer it. Check kubectl describe pod for the exact message. A timeout usually means the app has not started or the endpoint is slow, while a 404 or connection refused usually means the wrong path or port. If the application genuinely takes a long time to boot, the fix is a startup probe rather than a larger initialDelaySeconds, because a startup probe protects the boot window without weakening detection afterwards. Test the endpoint from inside the Pod with kubectl exec -it <pod> -- wget -qO- localhost:8080/healthz.

Do I need all three probes?

Readiness, essentially always. It is what makes rolling updates safe and keeps traffic off Pods that cannot serve. Liveness only when a restart would genuinely fix a condition you can detect, such as a deadlock; otherwise it adds restarts to a struggling system without helping. Startup only when the application takes long enough to boot that liveness would kill it first. A common and reasonable configuration is readiness alone, with a startup probe added for slow-starting services.

Should my health check verify the database connection?

Not in a liveness probe. A shared dependency fails for every replica at once, so the entire fleet gets restarted simultaneously and a slow database becomes a complete outage caused by the health check itself. A readiness probe may check that the application is able to serve, including that its connection pool is initialised, since removing Pods from the Service is recoverable and does not restart anything. The safest pattern is a /healthz that checks only the process and a /ready that checks whether this instance can serve a request.

What do initialDelaySeconds and failureThreshold actually control?

initialDelaySeconds is how long the kubelet waits before the first probe, periodSeconds how often it repeats, timeoutSeconds how long it waits for a response, and failureThreshold how many consecutive failures trigger the action. Time to detection is roughly periodSeconds × failureThreshold. The default timeoutSeconds of 1 is worth changing. An endpoint that occasionally takes just over a second under load will fail intermittently, and probes get slower precisely when the system is busiest.

Why do my Pods get no traffic even though they are Running?

They are Running but not Ready, so they have been excluded from the Service's endpoints. kubectl get pods shows this in the READY column as 0/1, and kubectl describe pod gives the probe's failure message. Common causes are a readiness path that requires authentication and returns 401, a wrong port, or an initialDelaySeconds shorter than the application's real startup time. Confirm with kubectl get endpointslices -l kubernetes.io/service-name=<service>. An empty list means no Pod is currently ready.