Kubernetes node NotReady

NotReady means the control plane is no longer satisfied that the node can run workloads. After roughly five minutes it taints the node node.kubernetes.io/unreachable and begins evicting pods, so this becomes a capacity problem quickly.

The first question is always whether the node is unhealthy or merely unable to talk to the API server. Those look identical from kubectl and have completely different causes.

Read the conditions

kubectl describe node ip-10-0-1-42 | grep -A8 Conditions

Conditions:
  Type                 Status    LastHeartbeatTime      Reason
  NetworkUnavailable   False     Mon, 16 Sep 09:02:11   RouteCreated
  MemoryPressure       Unknown   Mon, 16 Sep 09:04:33   NodeStatusUnknown
  DiskPressure         Unknown   Mon, 16 Sep 09:04:33   NodeStatusUnknown
  Ready                Unknown   Mon, 16 Sep 09:04:33   NodeStatusUnknown

Unknown with NodeStatusUnknown means the kubelet has stopped reporting at all. The node may be perfectly healthy and simply unreachable. Check LastHeartbeatTime against now to see when it went quiet.

False with a specific reason means the kubelet is reporting and telling you what is wrong. That is much better, because the reason names the cause:

Cause 1: Disk pressure

The most common cause on a long-lived node, and the one that resolves itself least often.

kubectl describe node ip-10-0-1-42 | grep -i pressure

On the node:

df -h /var/lib/kubelet /var/lib/containerd
df -i /var/lib/containerd

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  100G   95G  1.2G  99% /

The kubelet starts evicting at 85% by default and stops scheduling well before that. Two things fill a node: container images accumulating over months, and container logs.

sudo crictl images | wc -l
sudo du -sh /var/lib/containerd /var/log/pods

sudo crictl rmi --prune          # remove unused images

The durable fixes are larger root volumes, log rotation configured in the container runtime, and kubelet image garbage collection thresholds set deliberately rather than left at defaults. Remember that inodes can exhaust while space remains, which df -i shows.

Cause 2: Memory pressure

kubectl describe node ip-10-0-1-42 | grep -A5 "Allocated resources"

Resource   Requests      Limits
cpu        3800m (95%)   6000m (150%)
memory     14Gi (92%)    28Gi (180%)

Limits far above requests is the setup for this. Pods are scheduled against requests, so a node can accept far more work than it can actually run when every pod bursts toward its limit, and the kubelet then evicts to protect itself.

Check whether the OOM killer has been active:

dmesg -T | grep -i "out of memory" | tail -5

The fix is honest requests, and --kube-reserved plus --system-reserved on the kubelet so the node keeps memory for itself. Pods without requests at all are the worst case, since the scheduler treats them as needing nothing.

Cause 3: The kubelet is not running

sudo systemctl status kubelet
sudo journalctl -u kubelet --no-pager -n 100

Look for the reason it stopped. Frequent causes:

• Expired certificates. kubelet.conf client certificates rotate automatically, but a node that was stopped for longer than the certificate lifetime comes back unable to authenticate.
• A bad configuration change. An invalid flag or config file prevents startup entirely.
• The node was out of memory and the kubelet itself was killed.

sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

Cause 4: The container runtime is down

Ready  False  KubeletNotReady  container runtime is down

sudo systemctl status containerd
sudo crictl info | head -20
sudo crictl ps

The kubelet cannot report ready without a working runtime. containerd failing usually shows in its own journal, and a full disk is a common underlying cause, which is why disk pressure and runtime failure often appear together.

Cause 5: CNI not ready

Ready  False  KubeletNotReady  container runtime network not ready:
NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

kubectl get pods -n kube-system -o wide | grep -E "aws-node|calico|cilium|flannel"
ls /etc/cni/net.d/

The CNI DaemonSet pod on that node has failed or was never scheduled. On EKS that is aws-node; check its logs for IAM or IP exhaustion problems. An empty /etc/cni/net.d/ means the plugin never wrote its configuration.

This is the normal state for a brand new node for the first minute or so. It is only a problem if it persists.

Is it just unreachable?

If conditions are Unknown, the node may be fine:

# From a machine that can reach the node
ssh ip-10-0-1-42 'sudo systemctl is-active kubelet && uptime'

# From the node, can it reach the API server?
curl -sk https://<api-server>:443/healthz

Causes: a security group or firewall change, a NAT gateway failure, a route table change, or the API server endpoint being unreachable from that subnet. In cloud environments a node that is Unknown while the instance is healthy is very often a networking change rather than anything on the node.

Recovering

Drain it first, so pods move rather than being evicted abruptly:

kubectl drain ip-10-0-1-42 --ignore-daemonsets --delete-emptydir-data --timeout=300s

If the drain hangs, a PodDisruptionBudget is blocking it. Check kubectl get pdb -A for one showing ALLOWED DISRUPTIONS: 0.

Then fix or replace:

# Fix in place
sudo systemctl restart kubelet containerd

# Or replace it, which is usually faster and cleaner
kubectl delete node ip-10-0-1-42
aws autoscaling terminate-instance-in-auto-scaling-group \
  --instance-id i-0abc123 --should-decrement-desired-capacity

Replacing is usually the right call. A node that filled its disk once will do so again, and an Auto Scaling group brings up a clean one in minutes. Investigate the cause on the replacement, not on the casualty.

kubectl uncordon ip-10-0-1-42     # if you fixed it in place

Preventing it

Alarm on node conditions, not just node count. kube_node_status_condition in kube-state-metrics gives you DiskPressure and MemoryPressure before NotReady.

Set requests honestly, and set --kube-reserved and --system-reserved.

Rotate container logs and configure image garbage collection.

Replace nodes regularly. A fleet where no node is older than a few weeks does not accumulate the conditions that cause this, which is one of the better arguments for Karpenter's consolidation or a scheduled instance refresh.

A checklist

1. kubectl describe node <n> | grep -A8 Conditions.
2. Unknown → the node is not reporting; check network and the kubelet.
3. DiskPressure: True → df -h and df -i; prune images.
4. MemoryPressure: True → compare requests against limits; check dmesg.
5. systemctl status kubelet and its journal.
6. KubeletNotReady with a runtime message → systemctl status containerd.
7. CNI message → check the CNI DaemonSet pod on that node.
8. kubectl drain before doing anything disruptive.
9. Prefer replacing the node to repairing it.

---

Frequently Asked Questions

What does node NotReady mean in Kubernetes?

The control plane no longer believes the node can run workloads, either because the kubelet reported a problem or because it stopped reporting at all. After about five minutes the node is tainted node.kubernetes.io/unreachable and its pods begin being evicted, so capacity is affected. kubectl describe node shows the conditions, and whether they read False with a reason or Unknown tells you immediately which of those two situations you are in.

What is the difference between a node condition of False and Unknown?

False with a specific reason means the kubelet is alive and actively reporting a problem, disk pressure, memory pressure, a runtime or CNI failure, which is comparatively good news because the reason names the cause. Unknown with NodeStatusUnknown means no heartbeat has arrived, so the node may be entirely healthy and simply unable to reach the API server. Compare LastHeartbeatTime with the current time to see when it went quiet.

Why is my node showing DiskPressure?

The kubelet's root or image filesystem has crossed its eviction threshold, 85% by default. On a long-lived node the two things that fill it are accumulated container images and container logs. Check df -h /var/lib/containerd and also df -i, since inodes can exhaust while space remains. crictl rmi --prune reclaims unused images immediately; the durable fixes are log rotation, deliberate image garbage collection thresholds, and larger root volumes.

Why does a node show MemoryPressure when pods have limits set?

Because scheduling uses requests, not limits. A node can accept far more pods than it can actually run if limits are set well above requests, and when several pods burst toward their limits simultaneously the kubelet evicts to protect itself. Compare allocated requests against limits in kubectl describe node. Set requests close to real usage, configure --kube-reserved and --system-reserved, and treat pods with no requests at all as the worst offenders.

Should I fix a NotReady node or replace it?

Replace it, in most cases. A node that filled its disk or exhausted memory once will do so again, and an Auto Scaling group produces a clean replacement in minutes. Investigating on the replacement is safer than experimenting on a node that is currently carrying workloads. Drain first with kubectl drain --ignore-daemonsets so pods move gracefully, then delete the node object and terminate the instance. Repair in place is worth it mainly when the cause is a configuration change you can reverse.

Why is a brand new node NotReady with a CNI error?

Because the CNI plugin has not finished initialising. Until the network plugin writes its configuration to /etc/cni/net.d/ and reports ready, the kubelet cannot mark the node ready, so NetworkPluginNotReady is the normal state for the first minute or so after a node joins. It is only a problem if it persists, at which point check the CNI DaemonSet pod on that node, which on EKS means aws-node and often turns out to be an IAM permission or IP exhaustion issue.