Readiness probe failed: HTTP probe failed with statuscode: 503

A failing readiness probe does not restart anything. It removes the Pod from the Endpoints of every Service that selects it, so traffic stops arriving. The Pod stays Running with READY 0/1, which is why this often presents as "the deployment succeeded but the site is down".

A failing liveness probe is louder: the kubelet kills the container and restarts it. Repeatedly, that becomes a CrashLoopBackOff caused entirely by the probe.

Knowing which one you have is the first step.

kubectl get pod my-pod

NAME     READY   STATUS    RESTARTS   AGE
my-pod   0/1     Running   0          3m

0/1 with 0 restarts is readiness. Climbing restarts is liveness.

Read the exact failure

kubectl describe pod my-pod

The three messages you will see mean quite different things:

The first is usually honest: your application's own health endpoint is reporting a problem, often a database it cannot reach. The second is usually a configuration mistake. The third is usually a timeout set too tight, or an app that is genuinely hanging.

Test the endpoint from inside the Pod

This separates "the app is broken" from "the probe is wrong" in one command:

kubectl exec my-pod -- curl -sS -o /dev/null -w '%{http_code}\n' \
  http://localhost:8080/healthz

200

A 200 while the probe fails means the probe is not testing what you think. Compare it against the spec:

kubectl get pod my-pod -o jsonpath='{.spec.containers[0].readinessProbe}' | jq

{
  "httpGet": { "path": "/health", "port": 8000 },
  "initialDelaySeconds": 3,
  "periodSeconds": 10
}

Path /health against an app serving /healthz, port 8000 against an app on 8080. Either of those produces a failing probe on a perfectly healthy container.

If the container has no curl, use wget -qO-, or a shell redirect as a last resort:

kubectl exec my-pod -- sh -c 'exec 3<>/dev/tcp/localhost/8080 && echo open'

Fix 1: The probe points at the wrong port or path

The most common cause, and worth checking before anything else. Note that a probe's port can name a container port rather than a number, which is more robust:

ports:
  - name: http
    containerPort: 8080
readinessProbe:
  httpGet:
    path: /healthz
    port: http

Now renaming the port in one place cannot desynchronise the probe.

Fix 2: The app starts slower than the probe allows

A JVM or a service that warms a cache can take a minute. With initialDelaySeconds: 3 the liveness probe starts killing it before it has ever finished starting, and it never comes up. The restart count climbs and the logs show the app beginning to boot over and over.

The right answer is a startup probe, not a larger initialDelaySeconds:

startupProbe:
  httpGet:
    path: /healthz
    port: http
  failureThreshold: 30
  periodSeconds: 10          # allows up to 300s to start

livenessProbe:
  httpGet:
    path: /healthz
    port: http
  periodSeconds: 10
  failureThreshold: 3        # once started, react within ~30s

Liveness and readiness are both suspended until the startup probe passes once. You get a long grace period at boot and a tight check afterwards, instead of choosing between them.

Inflating initialDelaySeconds instead gives you a slow reaction to real failures forever, in exchange for surviving a slow boot once.

Fix 3: Timeout too tight

timeoutSeconds defaults to 1 second. A health endpoint that checks a database can easily exceed that under load, and then the probe fails for every replica at once, which takes the whole Service out precisely when it is busiest.

readinessProbe:
  httpGet:
    path: /healthz
    port: http
  timeoutSeconds: 5
  periodSeconds: 10
  failureThreshold: 3

This is a genuinely common cause of self-inflicted outages under load. If your probe does real work, give it room.

Liveness and readiness should not be the same endpoint

Pointing both at one endpoint that checks the database creates a failure mode where a slow database restarts every one of your Pods, which does nothing to fix the database and removes whatever capacity was still serving cached responses.

A reasonable split:

• Liveness: is this process wedged? Cheap, no dependencies. Return 200 if the event loop is turning.
• Readiness: can this instance serve a request right now? May check dependencies.

livenessProbe:
  httpGet: { path: /livez, port: http }
readinessProbe:
  httpGet: { path: /readyz, port: http }

/livez returns 200 unless the process is broken. /readyz can return 503 while the database is unreachable, which drains traffic without restarting anything.

The 503 that is telling the truth

If the endpoint really returns 503, read the application logs rather than tuning the probe:

kubectl logs my-pod --tail=50

Frameworks with built-in health endpoints, Spring Boot Actuator among them, aggregate dependency checks and return 503 when any is down. The probe is working correctly; something it depends on is not.

A checklist

1. kubectl get pod. 0/1 Running → readiness. Climbing restarts → liveness.
2. kubectl describe pod and read the exact message.
3. connection refused → wrong port, or the app binds 127.0.0.1 instead of 0.0.0.0.
4. Curl the endpoint from inside the Pod. 200 there → the probe is misconfigured.
5. context deadline exceeded → raise timeoutSeconds above the 1 second default.
6. Slow boot → add a startupProbe, do not inflate initialDelaySeconds.
7. Genuine 503 → read the app logs; a dependency is down.
8. Use separate /livez and /readyz so a slow dependency drains traffic instead of restarting Pods.

---

Frequently Asked Questions

What is the difference between a readiness and a liveness probe failure?

A failing readiness probe removes the Pod from the Endpoints of its Services, so traffic stops reaching it, but nothing is restarted. The Pod sits as Running with READY 0/1. A failing liveness probe makes the kubelet kill and restart the container, so you see the restart count climbing and, eventually, CrashLoopBackOff. Readiness answers "can this instance serve a request right now"; liveness answers "is this process wedged and in need of a restart".

Why does the probe fail when curling the endpoint works?

Because the probe is usually testing something slightly different. Compare the probe spec against what you curled: a path of /health against an app serving /healthz, or port 8000 against an app on 8080, both produce a failing probe on a healthy container. The other common cause is an application bound to 127.0.0.1 rather than 0.0.0.0: curling from inside the Pod as localhost works, while the kubelet probing the Pod IP gets a refused connection.

How do I fix a probe that kills my app before it finishes starting?

Add a startupProbe rather than increasing initialDelaySeconds. Liveness and readiness are both suspended until the startup probe succeeds once, so you can allow several minutes for boot with a high failureThreshold while keeping a tight failureThreshold on liveness afterwards. Raising initialDelaySeconds instead buys the same grace period at boot but permanently slows how quickly Kubernetes reacts to a genuine failure later.

What does "context deadline exceeded" mean on a probe?

The kubelet opened the connection and the application did not respond within timeoutSeconds, which defaults to just 1 second. A health endpoint that touches a database or another service can easily exceed that under load. The dangerous part is that it tends to fail for every replica simultaneously, taking the whole Service out exactly when it is busiest. If your health check does real work, set timeoutSeconds to something realistic such as 5, and consider making the readiness check cheaper.

Should liveness and readiness use the same endpoint?

No. If both point at an endpoint that checks the database, a slow database restarts every Pod you have, which does not help the database and destroys whatever capacity was still serving. Keep liveness cheap and dependency-free, so it only fails when the process itself is wedged, and let readiness check dependencies so that an unhealthy instance is drained of traffic without being killed. Two endpoints, commonly /livez and /readyz, are worth the small amount of extra code.