AWS EKS Node Groups

EKS Node Groups

The control plane is AWS's problem. The nodes are yours, and how you run them determines your cost, your upgrade pain and how quickly pods get scheduled.

Three ways to run workers

Managed node groups are the default choice. AWS handles the Auto Scaling group, the AMI, and a graceful drain-and-replace upgrade.

Self-managed is for what managed cannot express: a custom AMI with specific kernel settings, unusual GPU configurations, or Windows in older setups.

Fargate removes nodes entirely, and its limitations are what decide it. No DaemonSets, so any log shipper, security agent or service mesh sidecar delivered that way does not run. No privileged containers, no host networking, and a maximum of 4 vCPU and 30 GB per pod. It suits bursty, isolated workloads; it rarely suits a general-purpose cluster.

A managed node group

eksctl create nodegroup --config-file=- <<'EOF'
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: prod
  region: eu-west-1
managedNodeGroups:
  - name: general-v2
    instanceTypes: ["m6i.large", "m6a.large", "m5.large"]
    minSize: 2
    maxSize: 20
    desiredCapacity: 3
    privateNetworking: true
    volumeSize: 100
    volumeType: gp3
    amiFamily: AmazonLinux2023
    labels:
      workload: general
    updateConfig:
      maxUnavailablePercentage: 25
    iam:
      withAddonPolicies:
        cloudWatch: true
EOF

List several instance types. It widens the pool the group can draw from, which matters for capacity during a busy period and enormously for Spot.

Include a version in the name. Node group configuration is largely immutable. Instance types, AMI family and subnets cannot be changed after creation, so "upgrading" often means creating general-v2, migrating, and deleting general. A name without a version makes that awkward.

Taints and labels

Labels attract pods; taints repel them. Together they keep workloads where they belong:

managedNodeGroups:
  - name: gpu
    instanceTypes: ["g5.xlarge"]
    labels:
      workload: ml
      nvidia.com/gpu: "true"
    taints:
      - key: nvidia.com/gpu
        value: "true"
        effect: NoSchedule

The taint is what stops ordinary pods landing on expensive GPU instances. Without it the scheduler treats them as general capacity and you pay GPU rates to run a web service. A mistake that shows up on the bill rather than in any alert.

# In the workload
nodeSelector:
  workload: ml
tolerations:
  - key: nvidia.com/gpu
    operator: Equal
    value: "true"
    effect: NoSchedule

Karpenter over Cluster Autoscaler

For node scaling there are two options, and they are not close any more.

Cluster Autoscaler adjusts the desired capacity of existing node groups. It can only launch instance types those groups already define, needs a group per instance shape, and takes several minutes.

Karpenter watches unschedulable pods and launches instances chosen to fit them, right-sizing automatically, consolidating underused nodes, and provisioning in under a minute.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s

Rather than enumerating instance types, you describe acceptable characteristics and let Karpenter pick. consolidationPolicy then actively repacks pods onto fewer nodes and terminates the remainder, which is where most of the saving comes from. Cluster Autoscaler only scales down nodes that are already nearly empty.

Keep a small managed node group for Karpenter itself and CoreDNS. Karpenter cannot provision the nodes it runs on.

Spot instances

Spot is where cluster cost drops by 60 to 90%, and it is safe for stateless workloads given three things.

Diversify. Accept many instance types and families so an interruption in one pool is absorbed elsewhere. A Spot node group pinned to a single type is what gives Spot its reputation.

Handle interruptions. AWS gives a two-minute warning; something must cordon and drain the node. Karpenter handles this natively via an SQS queue; with managed node groups, run the AWS Node Termination Handler.

Set PodDisruptionBudgets, or a drain can evict every replica at once:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api

Mix capacity types rather than going all-Spot: a baseline of on-demand for the workloads that must not move, Spot for everything else.

Upgrading nodes

aws eks update-nodegroup-version --cluster-name prod \
  --nodegroup-name general --kubernetes-version 1.32

AWS cordons each node, drains it respecting PodDisruptionBudgets, waits, then terminates it.

That respect for PDBs is a double edge. A PDB that can never be satisfied, minAvailable: 1 on a single-replica Deployment, blocks the drain indefinitely, and the upgrade hangs for hours before failing:

NodeCreationFailure: Couldn't proceed with upgrade process as the node
group has pods that cannot be evicted

Before any upgrade, check for PDBs that cannot be met:

kubectl get pdb -A

NAMESPACE   NAME   MIN AVAILABLE   ALLOWED DISRUPTIONS
default     api    2               1
payments    db     1               0        ← blocks drains

ALLOWED DISRUPTIONS: 0 is the row that will stop your upgrade. --force skips PDBs entirely, which gets the upgrade moving and causes the outage the PDB existed to prevent.

Set maxUnavailablePercentage to control the pace: 25% replaces a quarter of the group at a time, which is much faster than the default of one node while still being gradual.

Diagnosing a node that will not join

A node that launches but never appears in kubectl get nodes is nearly always one of four things:

kubectl get nodes
aws eks describe-nodegroup --cluster-name prod --nodegroup-name general \
  --query 'nodegroup.health.issues'

• IAM: the node role is missing AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly or AmazonEKS_CNI_Policy.
• Networking: a private subnet with no NAT gateway and no VPC endpoints cannot reach the EKS API or ECR.
• Security groups: the cluster and node groups must permit traffic in both directions.
• User data: a custom launch template whose user data does not call the bootstrap script.

describe-nodegroup's health.issues usually names it directly, and is faster than reading instance logs.

For a node that joined and later went NotReady, look at the kubelet:

kubectl describe node ip-10-0-1-42.ec2.internal | grep -A10 Conditions

DiskPressure is common and usually means image accumulation. Set imageGCHighThresholdPercent or use larger root volumes.

Common errors

NodeCreationFailure: Instances failed to join the kubernetes cluster

IAM, networking, security groups or bootstrap user data.

NodeCreationFailure: Couldn't proceed with upgrade process as the node group
has pods that cannot be evicted

An unsatisfiable PodDisruptionBudget. Fix the PDB or the replica count.

0/5 nodes are available: 5 node(s) had untolerated taint

The pod lacks a toleration for a tainted node group.

Nodes at Ready but pods stay Pending. Often CNI IP exhaustion rather than a node problem.

Quick reference

---

Key takeaways

• Managed node groups are the default. Fargate's lack of DaemonSets rules it out for most clusters.
• List several instance types: essential for Spot, useful for capacity generally.
• Taint expensive node groups, or ordinary pods will schedule onto GPU instances.
• Prefer Karpenter to Cluster Autoscaler: faster, right-sizing, and it consolidates.
• Spot needs diversification, interruption handling and PodDisruptionBudgets.
• Check kubectl get pdb -A before upgrading: ALLOWED DISRUPTIONS: 0 blocks the drain.
• --force skips PDBs and causes the outage they were preventing.
• describe-nodegroup --query 'nodegroup.health.issues' names most join failures directly.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between managed node groups, self-managed nodes and Fargate?

Managed node groups have AWS run the Auto Scaling group, supply the AMI and orchestrate graceful upgrades. The right default. Self-managed nodes give you full control of the AMI, user data and lifecycle, which you need for custom kernels or unusual hardware. Fargate removes nodes entirely and charges per pod, but supports no DaemonSets, no privileged containers and no host networking, and caps a pod at 4 vCPU and 30 GB. That DaemonSet restriction alone rules Fargate out for most clusters running log shippers or security agents.

Should I use Karpenter or Cluster Autoscaler?

Karpenter, on new clusters. Cluster Autoscaler only adjusts the size of node groups you have already defined, so it can only launch instance types those groups specify and needs a group per shape. Karpenter looks at pending pods and provisions instances that fit them, in under a minute, choosing from broad requirements rather than a fixed list, and it consolidates underused nodes, which is where most of the cost saving comes from. Keep a small managed node group for Karpenter itself and CoreDNS.

How do I run Spot instances safely on EKS?

Three things together. Diversify across many instance types and families so one pool's interruption is absorbed elsewhere. A single-type Spot group is what earns Spot its bad reputation. Handle the two-minute interruption notice by cordoning and draining, which Karpenter does natively and managed node groups need the AWS Node Termination Handler for. And set PodDisruptionBudgets so a drain cannot evict every replica at once. Keep a baseline of on-demand capacity for anything that must not move.

Why won't my EKS nodes join the cluster?

Four causes cover nearly all of it, and aws eks describe-nodegroup --query 'nodegroup.health.issues' usually names the right one. The node IAM role may be missing AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly or AmazonEKS_CNI_Policy. A private subnet may have no NAT gateway or VPC endpoints, leaving nodes unable to reach the EKS API or ECR. Security groups may not permit cluster-to-node traffic both ways. Or a custom launch template's user data may not invoke the bootstrap script.

What is the difference between a taint and a label on a node group?

A label is descriptive. It lets a pod request that node through a nodeSelector or affinity rule, and does nothing otherwise. A taint is restrictive: it prevents any pod without a matching toleration from being scheduled there. Labels alone on an expensive node group are not enough, because the scheduler sees untainted capacity and uses it, so ordinary pods land on GPU instances. Use both: a taint to keep others out, and a label plus nodeSelector to draw the intended workload in.

How do I upgrade EKS nodes without downtime?

Use update-nodegroup-version, which cordons, drains respecting PodDisruptionBudgets, and replaces each node. Before starting, run kubectl get pdb -A and fix any budget showing ALLOWED DISRUPTIONS: 0, since that blocks the drain indefinitely and the upgrade eventually fails. Set maxUnavailablePercentage to around 25 so the rollout is gradual but not glacial. Avoid --force: it ignores PDBs and produces the outage they were protecting against.