EKS nodes not joining the cluster

The instances launch and never register. Work through these in order; the first two account for most cases.

aws eks describe-nodegroup --cluster-name prod --nodegroup-name workers \
  --query 'nodegroup.health.issues'

[{ "code": "NodeCreationFailure",
   "message": "Instances failed to join the kubernetes cluster" }]

1. The node IAM role

Three managed policies are required and all three matter:

aws iam list-attached-role-policies --role-name eks-node-role \
  --query 'AttachedPolicies[].PolicyName'

AmazonEKSWorkerNodePolicy
AmazonEKS_CNI_Policy
AmazonEC2ContainerRegistryReadOnly

The CNI one is the frequently missed member, and it produces a different symptom: the node appears and never becomes Ready.

2. Network path to the API server

A node must reach the cluster endpoint to register.

Private subnet with no NAT gateway:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-0private1" \
  --query 'RouteTables[].Routes[?DestinationCidrBlock==`0.0.0.0/0`]'

No default route means no path out, and the bootstrap hangs until it times out.

If you are using VPC endpoints instead of NAT, nodes need several: ec2, ecr.api, ecr.dkr, the S3 gateway, sts, and logs. Missing sts is a common one, because the bootstrap calls STS to get a token and fails with failed to get token.

Private cluster endpoint:

aws eks describe-cluster --name prod \
  --query 'cluster.resourcesVpcConfig.{public:endpointPublicAccess,private:endpointPrivateAccess}'

With endpointPublicAccess: false, nodes must reach the private endpoint, which requires the cluster security group to allow them and DNS resolution inside the VPC.

3. Security groups

Nodes and the control plane must talk both ways:

aws eks describe-cluster --name prod \
  --query 'cluster.resourcesVpcConfig.clusterSecurityGroupId'

Required:

• Node security group: outbound 443 to the cluster security group
• Cluster security group: inbound 443 from the node security group
• Node to node: all traffic within the node security group, for Pod networking and DNS

The EKS-managed cluster security group covers the first two by default. A custom security group on a launch template frequently does not, and that is where this breaks.

4. Subnet tags

The VPC CNI and load balancer controller rely on tags:

aws ec2 describe-subnets --subnet-ids subnet-0private1 \
  --query 'Subnets[].Tags'

Missing the cluster tag produces Ec2SubnetInvalidConfiguration rather than a silent failure, which is at least clear.

Also check the subnet has free addresses. The VPC CNI assigns a real VPC IP to every Pod, so a /27 runs out quickly and new nodes cannot allocate ENIs:

aws ec2 describe-subnets --subnet-ids subnet-0private1 \
  --query 'Subnets[].AvailableIpAddressCount'

5. The AMI and bootstrap

A custom launch template that overrides user data must still call the bootstrap script:

#!/bin/bash
/etc/eks/bootstrap.sh prod

Replacing user data without that line gives you an instance that boots and never joins. For AL2023 nodes the mechanism is different, a NodeConfig YAML in user data rather than the shell script, and mixing the two is a common error after an AMI upgrade.

Check the AMI matches the cluster version. A node AMI more than one minor version behind the control plane is unsupported and may not register.

Read the instance

aws ssm start-session --target i-0abc123

sudo journalctl -u kubelet -n 100 --no-pager
sudo cat /var/log/cloud-init-output.log | tail -50

This is the fastest route when the health message is generic. The kubelet log usually names the exact failure: a token error, a connection timeout to the endpoint, or a missing configuration file.

SSM needs AmazonSSMManagedInstanceCore on the node role, which is worth attaching to every node group for exactly this reason.

If aws-auth is still in use

On a CONFIG_MAP cluster the node role must be mapped, or nodes authenticate as nobody:

mapRoles: |
  - rolearn: arn:aws:iam::123456789012:role/eks-node-role
    username: system:node:{{EC2PrivateDNSName}}
    groups: [system:bootstrappers, system:nodes]

Managed node groups add this automatically. Self-managed nodes and Karpenter do not, and a deleted entry stops every future node joining while existing ones carry on working, which makes it look like a capacity problem.

A checklist

1. describe-nodegroup --query 'nodegroup.health.issues' for the code.
2. Node role needs all three managed policies. The CNI one is the usual omission.
3. Private subnet → NAT gateway, or VPC endpoints including sts and the S3 gateway.
4. Security groups: 443 both ways between nodes and the cluster SG, plus node to node.
5. Subnet tagged kubernetes.io/cluster/<name>.
6. AvailableIpAddressCount on the subnet. The CNI uses real VPC IPs.
7. Custom launch template → the bootstrap call must still be there.
8. aws ssm start-session, then journalctl -u kubelet.

---

Frequently Asked Questions

Which IAM policies does an EKS node role need?

Three: AmazonEKSWorkerNodePolicy so the node can describe the cluster and register, AmazonEKS_CNI_Policy so the VPC CNI can attach ENIs and assign Pod IPs, and AmazonEC2ContainerRegistryReadOnly so it can pull images. The CNI policy is the one most often forgotten, and it produces a distinctive symptom: the node registers and appears in kubectl get nodes but stays NotReady, because networking never comes up. Attaching AmazonSSMManagedInstanceCore as well is worth it for debugging.

Why do nodes in a private subnet fail to join?

Because they have no route to the EKS API endpoint. Either add a NAT gateway with a default route, or create VPC endpoints. The endpoint list is longer than people expect: ec2, ecr.api, ecr.dkr, the S3 gateway endpoint for image layers, sts because the bootstrap exchanges an STS token, and logs for CloudWatch. A missing sts endpoint produces failed to get token, which is a useful clue when you see it.

What happens if the subnet is not tagged?

The kubernetes.io/cluster/<cluster-name> tag is used for resource discovery, and without it EKS reports Ec2SubnetInvalidConfiguration on the node group. The kubernetes.io/role/elb and kubernetes.io/role/internal-elb tags are separate and affect load balancer placement rather than node joining, so a cluster with healthy nodes can still fail to provision a LoadBalancer Service if those are missing.

Why did nodes stop joining when existing ones are fine?

On a cluster still using the aws-auth ConfigMap, the node role mapping is what lets new nodes authenticate. Deleting or corrupting that entry stops every future node joining while running nodes continue working, because they are already authenticated. The symptom looks like a capacity or scaling problem. Managed node groups maintain the entry automatically; self-managed nodes and Karpenter rely on you keeping it, which is one more reason to move to access entries.

How do I see why a specific instance failed to bootstrap?

Connect with aws ssm start-session --target <instance-id>, which needs AmazonSSMManagedInstanceCore on the node role, then read journalctl -u kubelet -n 100 and /var/log/cloud-init-output.log. The kubelet log names the actual failure rather than the generic NodeCreationFailure the node group reports: a token error points at STS or IAM, a connection timeout at the network path, and a missing config file at a broken launch template.