context deadline exceeded

context deadline exceeded is Go's standard timeout error. Kubernetes is written in Go, so almost every component can emit it, and on its own it tells you only that something waited and gave up.

The prefix is the diagnosis. Read it before anything else.

Admission webhooks: the one that breaks everything

Internal error occurred: failed calling webhook "vpolicy.kb.io":
Post "https://webhook-service.default.svc:443/validate?timeout=10s":
context deadline exceeded

Every create and update of a matching resource goes through the webhook. If its backend is down, the API server waits timeoutSeconds and then applies failurePolicy. With failurePolicy: Fail, the request is rejected, and nothing matching that webhook can be created cluster-wide.

This is the classic way a cluster becomes unable to schedule anything: the webhook's own Pods get evicted, and now they cannot be recreated because the webhook that must admit them is down.

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get endpoints -n cert-manager cert-manager-webhook

Empty ENDPOINTS confirms it.

To break the deadlock, delete the webhook configuration, fix the backend, then reinstall it:

kubectl get validatingwebhookconfiguration cert-manager-webhook -o yaml > /tmp/webhook.yaml
kubectl delete validatingwebhookconfiguration cert-manager-webhook
# recover the backend, then:
kubectl apply -f /tmp/webhook.yaml

Save it first. Deleting without a copy loses the CA bundle, which is not always easy to regenerate.

Two things worth designing for:

Scope the webhook with namespaceSelector so it never intercepts kube-system or its own namespace:

namespaceSelector:
  matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values: [kube-system, cert-manager]

Use failurePolicy: Ignore for anything non-critical. A policy webhook that annotates Pods should not be able to stop the cluster. Reserve Fail for genuine security controls where allowing an unchecked object through is worse than an outage.

API server or etcd

A bare timeout from kubectl on many different commands points at the control plane:

kubectl get --raw /readyz?verbose
kubectl get --raw /metrics | grep etcd_request_duration_seconds_bucket | tail -5

etcd is the usual constraint, and it is disk-latency sensitive far more than CPU. fsync durations above about 25ms produce visible cluster-wide slowness. Common causes: etcd on network-attached storage rather than local NVMe, a large number of Events, or oversized objects.

kubectl get events -A --no-headers | wc -l

Tens of thousands of Events is a real load. They expire after an hour by default, so a persistently large count means something is generating them fast.

Also check for very large objects. A ConfigMap near the 1MiB limit being read frequently by a controller is a surprisingly effective way to slow etcd down.

CSI and device plugin gRPC

rpc error: code = DeadlineExceeded desc = context deadline exceeded

This is a gRPC call to a driver that did not answer. Look at the driver, not Kubernetes:

kubectl -n kube-system logs -l app=ebs-csi-controller -c ebs-plugin --tail=50

Cloud API rate limiting is a frequent cause on large clusters. Many volumes being attached at once exhausts the provider's request quota and calls start timing out, which looks like a Kubernetes fault and is not.

Probes

Covered in depth on the readiness probe page, but the short version: timeoutSeconds defaults to 1 second, which a health endpoint that touches a database will exceed under load. It then fails for every replica at once.

readinessProbe:
  timeoutSeconds: 5

kubectl itself

If only your machine is affected:

time kubectl get nodes
kubectl get nodes --request-timeout=30s
kubectl get nodes -v=6         # shows round-trip timings

A private API endpoint reached over a VPN, or a long path to another region, produces timeouts that look like cluster problems and are not. -v=6 prints the request duration, which settles it quickly.

For genuinely large collections, paginate rather than raising the timeout:

kubectl get pods -A --chunk-size=500

A checklist

1. Read the prefix. It names the component.
2. failed calling webhook → check that webhook's Service endpoints.
3. Webhook backend down and blocking recovery → save and delete the configuration.
4. Scope webhooks with namespaceSelector and prefer failurePolicy: Ignore.
5. Cluster-wide slowness → /readyz?verbose and etcd request duration metrics.
6. Count Events. Tens of thousands is load worth fixing.
7. rpc error → the CSI or device plugin logs, and cloud API rate limits.
8. Only your machine → kubectl -v=6 and check the network path.

---

Frequently Asked Questions

What does "context deadline exceeded" actually mean?

It is Go's generic timeout error, produced when an operation carrying a deadline runs out of time. Kubernetes components are written in Go, so the API server, kubelet, controllers and CSI drivers can all emit it, and the phrase itself carries no information about what was slow. The text before it does: failed calling webhook is admission, Liveness probe failed is the kubelet, and rpc error: code = DeadlineExceeded is a gRPC call to a plugin. Identify the component before investigating anything.

Why did a broken admission webhook stop me creating any Pods?

Because every matching create and update goes through the webhook, and with failurePolicy: Fail the API server rejects the request when the webhook does not answer. If the webhook's own Pods are down, you get a deadlock: the Pods cannot be recreated because the webhook that must admit them is unavailable. The way out is to save the webhook configuration to a file, delete it, recover the backend and reapply it. Scoping webhooks with a namespaceSelector that excludes kube-system and their own namespace prevents the deadlock forming.

How do I tell whether the API server or my network is slow?

Run kubectl get nodes -v=6, which prints the round-trip duration for each request. If that is fast while the cluster is otherwise unhappy, your connection is fine. For the control plane, kubectl get --raw /readyz?verbose reports each component's health, and the etcd_request_duration_seconds histogram in /metrics shows whether etcd is the constraint. etcd is unusually sensitive to disk latency, so fsync times above roughly 25 milliseconds produce cluster-wide slowness even when CPU looks fine.

Can too many Events slow down my cluster?

Yes. Events are stored in etcd like any other object, and a cluster generating them rapidly, typically from a Pod crash-looping at scale or a controller reconciling in a tight loop, puts real write pressure on it. They expire after an hour by default, so a persistently high count from kubectl get events -A --no-headers | wc -l means something is producing them continuously. Fixing the source matters more than tuning retention, since the write load is the problem rather than the storage.

Should I set failurePolicy to Ignore on all my webhooks?

No, but you should default to it and make Fail a deliberate choice. Fail is correct for a security control where admitting an unchecked object is worse than refusing the request, such as a policy that blocks privileged containers. For anything that annotates, defaults or labels, Ignore means a webhook outage degrades a convenience rather than stopping the cluster. Whichever you choose, keep timeoutSeconds low, since the API server waits the full timeout on every matching request before applying the policy.