0/3 nodes are available: node(s) had untolerated taint

node(s) had untolerated taint means the scheduler did find nodes, and every one of them pushed your Pod away. A taint is a node saying "nothing runs here unless it explicitly says it can tolerate this". Your Pod does not say that.

This is different from Insufficient cpu, which is a capacity problem. Here there may be plenty of room; the nodes are simply refusing.

Read the taint first

The scheduler tells you exactly which taint blocked it. Do not guess:

kubectl describe pod my-pod

Events:
  Type     Reason            Message
  ----     ------            -------
  Warning  FailedScheduling  0/3 nodes are available: 3 node(s) had untolerated
                             taint {node-role.kubernetes.io/control-plane: }.

The part in braces is a key: value pair. Here the key is node-role.kubernetes.io/control-plane and the value is empty. That pair is what a toleration has to match.

A cluster with mixed taints gives you a breakdown:

0/5 nodes are available: 2 node(s) had untolerated taint
{node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint
{dedicated: gpu}.

Two separate problems in one message. Two nodes are unhealthy and three are reserved.

See all of them at once:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

NAME       TAINTS
cp-1       [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
work-1     <none>
work-2     [map[effect:NoSchedule key:dedicated value:gpu]]

The single-node case, which is most of them

If you built a cluster with kubeadm and never joined a worker, every node is a control-plane node and carries node-role.kubernetes.io/control-plane:NoSchedule. Nothing you deploy will ever schedule.

This is by design: control-plane nodes are meant to stay clear of workloads. On a one-node lab cluster that design is in your way, so remove it deliberately:

kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-

The trailing - is the removal syntax and it is easy to miss. Without it you are adding a taint, not removing one.

Older clusters use node-role.kubernetes.io/master instead, and some carry both. Remove whichever kubectl describe node actually shows.

Do not do this on a real cluster. Add worker nodes instead.

The three effects

NoExecute is the one that causes an incident. Taint a node with it and running Pods are removed, not just kept away. That is exactly what Kubernetes does automatically when a node goes unhealthy, which is how a node problem turns into Pods moving.

Fix 1: Tolerate the taint

When the taint is correct and your Pod genuinely belongs there, add a toleration:

spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

For a taint with no value, such as the control-plane one, use Exists:

spec:
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

A toleration is permission, not preference. It lets the Pod land on a tainted node; it does not make the scheduler prefer one. If you want the Pod to actually go to the GPU nodes rather than merely be allowed to, pair the toleration with a nodeSelector or node affinity:

spec:
  nodeSelector:
    accelerator: nvidia-a100
  tolerations:
    - key: dedicated
      operator: Equal
      value: gpu
      effect: NoSchedule

Without the selector, a tolerating Pod will happily take a normal node and leave the expensive GPU capacity idle. This is the single most common mistake with taints.

Fix 2: Remove the taint

Only when the taint should not be there. Something automated may put it back.

# Remove a specific taint by key, value and effect
kubectl taint nodes work-2 dedicated=gpu:NoSchedule-

# Remove every taint with a given key, whatever its value
kubectl taint nodes work-2 dedicated-

If the taint reappears within a minute or two, something is re-applying it: a node pool configuration, a cloud provider setting, or a DaemonSet-managed agent. Fix it there, because the cluster will keep undoing you.

The taints Kubernetes adds by itself

These are never yours and they mean the node is unhealthy:

Tolerating these is almost always wrong. It tells the scheduler to place work on a node you have been told is broken. Fix the node instead.

unschedulable is the exception in one direction: it is what kubectl cordon sets, so if you cordoned a node during maintenance and forgot, kubectl uncordon <node> is the fix.

Why DaemonSets seem exempt

They are not exempt, they are tolerant. The DaemonSet controller adds tolerations for the standard node-condition taints automatically, because a log shipper or CNI agent needs to run on a node precisely when that node is in trouble. It is a good illustration of when tolerating a health taint is correct: for infrastructure that exists to observe or repair the node.

A checklist

1. kubectl describe pod <pod> and read the taint inside the braces.
2. kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints.
3. Single-node kubeadm cluster → remove the control-plane taint with the trailing -.
4. Taint is deliberate → add a matching toleration to the Pod spec.
5. Want the Pod to actually land there → add a nodeSelector too, not just the toleration.
6. Taint is node.kubernetes.io/* → the node is unhealthy, fix the node.
7. Taint comes back after removal → something is re-applying it, change that instead.

---

Frequently Asked Questions

What does "node(s) had untolerated taint" actually mean?

The scheduler found candidate nodes, and every one of them carries a taint your Pod has no matching toleration for, so it refused to place the Pod on any of them. It is a permissions problem rather than a capacity problem: there may be plenty of free CPU and memory. The scheduler names the offending taint in braces in the FailedScheduling event, for example {dedicated: gpu}, and that key and value are exactly what a toleration must match.

How do I remove the control-plane taint on a single-node cluster?

kubectl taint nodes --all node-role.kubernetes.io/control-plane:NoSchedule-. The trailing hyphen is the removal syntax and is easy to miss; without it you are adding the taint rather than removing it. Older clusters use node-role.kubernetes.io/master and some carry both, so check kubectl describe node for the exact key. This is appropriate on a lab or single-node cluster where there are no workers, and a bad idea on a real cluster, where the taint exists to keep workloads off the control plane.

What is the difference between NoSchedule and NoExecute?

NoSchedule stops new Pods landing on the node but leaves anything already running alone. NoExecute also evicts the Pods that are already there. That difference matters a great deal in production: applying a NoExecute taint to a busy node moves live workloads immediately. It is also how Kubernetes reacts to node problems automatically, adding node.kubernetes.io/unreachable:NoExecute when a kubelet stops reporting, which is why a node going quiet causes Pods to move.

Why does my Pod still not run on the GPU node after I added a toleration?

Because a toleration is permission, not preference. It allows the Pod onto a tainted node; it does not direct it there. The scheduler is still free to pick any node that fits, and it will often pick an ordinary one, leaving the expensive capacity idle. Pair the toleration with a nodeSelector or node affinity that matches a label on the GPU nodes. Taints keep the wrong workloads off; labels and selectors pull the right ones on. You generally need both.

Should I tolerate node.kubernetes.io/not-ready?

Almost never for an application. That taint means the node is unhealthy, and tolerating it tells the scheduler to place work somewhere Kubernetes has already flagged as broken. Fix the node instead: check kubectl describe node for the condition, and the kubelet logs on the host. The legitimate exception is infrastructure that must run on a sick node in order to observe or repair it, which is why the DaemonSet controller adds tolerations for the node-condition taints to its Pods automatically.