Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler adds and removes replicas to keep an observed metric near a target. More Pods, not bigger ones, that is the horizontal part, and it is what makes it suitable for stateless services.

It is one of the more genuinely useful controllers in Kubernetes, and also one people wire up once and never verify.

The formula

Everything the HPA does follows from one line:

desiredReplicas = ceil( currentReplicas × ( currentMetric / targetMetric ) )

Three replicas averaging 90% CPU against a 50% target gives ceil(3 × 1.8) = 6 replicas. The controller re-evaluates every 15 seconds by default.

Two consequences worth noticing. It is proportional, so a large overshoot produces a large jump rather than one replica at a time. And it depends on currentReplicas, which is why an HPA cannot scale up from zero. Anything times zero is zero. Scale-to-zero needs KEDA.

It requires resource requests

This is the single most common reason an HPA does nothing.

CPU and memory targets are expressed as a percentage of the request. A container with no CPU request has no denominator, so utilisation is undefined and the HPA reports <unknown> and refuses to act.

kubectl get hpa

NAME           REFERENCE                 TARGETS         MINPODS  MAXPODS  REPLICAS
payments-api   Deployment/payments-api   <unknown>/50%   2        10       2

<unknown> means one of exactly two things: metrics-server is not installed, or the target Pods have no resource requests.

kubectl top pods                # fails entirely if metrics-server is missing
kubectl get deployment payments-api -o jsonpath='{.spec.template.spec.containers[0].resources}'

A working HPA

Use autoscaling/v2. The older v1 supports CPU only.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payments-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payments-api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

kubectl apply -f hpa.yaml
kubectl get hpa payments-api -w

NAME           REFERENCE                 TARGETS    MINPODS  MAXPODS  REPLICAS
payments-api   Deployment/payments-api   34%/60%    2        20       2
payments-api   Deployment/payments-api   81%/60%    2        20       3
payments-api   Deployment/payments-api   58%/60%    2        20       3

Remove replicas from the Deployment manifest once an HPA manages it. Otherwise every kubectl apply resets the replica count to whatever the file says, and the HPA scales it back up moments later. A loop that looks like flapping and is really two controllers disagreeing. If you use GitOps, exclude the field from diffing.

Controlling how it scales

By default the HPA scales up quickly and down slowly. A deliberate asymmetry, since being briefly over-provisioned is much cheaper than being under-provisioned. The behavior block makes it explicit:

spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0      # react immediately
      policies:
        - type: Percent
          value: 100                     # at most double
          periodSeconds: 15
        - type: Pods
          value: 4                       # or at most +4
          periodSeconds: 15
      selectPolicy: Max                  # whichever allows more
    scaleDown:
      stabilizationWindowSeconds: 300    # wait 5 minutes of calm
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

stabilizationWindowSeconds on scale-down is the setting that matters most. It makes the HPA use the highest recommendation from the last N seconds, which prevents a brief dip in traffic from removing capacity that is needed again a minute later. The 300-second default is sensible; shortening it causes thrashing.

Beyond CPU

CPU is a poor proxy for load in plenty of services. An I/O-bound API may be saturated at 20% CPU. autoscaling/v2 supports better signals.

Memory, though be careful: many runtimes hold memory rather than returning it, so memory rarely falls after a spike and the HPA will not scale back down.

    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

Custom metrics from your application, via Prometheus Adapter. Requests per second is usually far more meaningful than CPU:

    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

External metrics such as queue depth, which is the correct signal for a worker pool:

    - type: External
      external:
        metric:
          name: sqs_messages_visible
          selector:
            matchLabels:
              queue: payments-jobs
        target:
          type: AverageValue
          averageValue: "30"

With several metrics listed, the HPA computes a desired replica count for each and takes the highest. That is the safe behaviour. Any one signal indicating pressure is enough to scale up, and every signal must be calm to scale down.

Nodes have to exist

An HPA that wants 20 replicas on a cluster with room for 12 leaves eight Pods Pending. Pod autoscaling and node autoscaling are separate concerns, and you generally need both:

• Cluster Autoscaler grows and shrinks node groups.
• Karpenter provisions right-sized instances directly, and responds faster.

Set maxReplicas to a number your cluster or your budget can actually absorb. It is a safety limit as much as a scaling parameter.

Diagnosing

kubectl describe hpa payments-api

Conditions:
  Type            Status  Reason               Message
  AbleToScale     True    ReadyForNewScale     recommended size matches current
  ScalingActive   True    ValidMetricFound     the HPA was able to compute a replica count
  ScalingLimited  True    TooManyReplicas      the desired count is above the maximum

Those three conditions answer most questions directly:

• ScalingActive: False: metrics unavailable. metrics-server missing, or no resource requests.
• ScalingLimited: True: it wants to scale further but minReplicas/maxReplicas is in the way.
• AbleToScale: False: it cannot modify the target, often a conflicting controller or a missing scale subresource.

kubectl get events --field-selector involvedObject.name=payments-api

What it will not do

• Scale from zero. The formula multiplies by current replicas. Use KEDA.
• Scale StatefulSets safely by default. It works mechanically, but adding a database replica is rarely just a replica count.
• Work alongside VPA on the same metric. VPA resizing Pods while HPA counts them on CPU causes both to fight. Use VPA for memory and HPA for CPU, or keep them apart.

KEDA is worth knowing about: it extends the HPA with scale-to-zero and dozens of event sources. Kafka lag, SQS depth, RabbitMQ, cron. For queue-driven workers it is usually the better tool.

Quick reference

---

Key takeaways

• desiredReplicas = ceil(current × (currentMetric / target)): proportional, and unable to start from zero.
• Resource requests are mandatory. Utilisation is a percentage of the request; with no request it is <unknown>.
• <unknown> means metrics-server is missing or requests are unset. Only those two.
• Remove replicas from a Deployment managed by an HPA, or the two controllers fight on every apply.
• stabilizationWindowSeconds on scale-down prevents thrashing. The 300s default is sensible.
• With several metrics, the highest recommendation wins.
• Pod autoscaling needs node autoscaling or the extra replicas sit Pending.

---

Check your understanding

---

Frequently Asked Questions

Why does my HPA show <unknown> for the target metric?

Two causes only. Either metrics-server is not installed, in which case kubectl top pods also fails, or the target Pods have no resource request for the metric being measured. A Utilization target is a percentage of the request, so without a request there is nothing to compute against. Check with kubectl get deployment <name> -o jsonpath='{.spec.template.spec.containers[0].resources}' and add requests if they are missing.

How does the HPA decide how many replicas to run?

It applies desiredReplicas = ceil(currentReplicas × (currentMetric / targetMetric)), recalculated every 15 seconds by default. Three replicas averaging 90% CPU against a 50% target gives six. Because the calculation is proportional it corrects large overshoots in one step rather than incrementing. It also explains why an HPA cannot scale up from zero: multiplying zero replicas by any ratio gives zero, which is why scale-to-zero requires KEDA.

Can the HPA scale on something other than CPU?

Yes, with autoscaling/v2. Memory works but is often a poor signal, since many runtimes retain memory rather than releasing it so the value never falls enough to scale back down. Custom metrics from your application, requests per second, for example, are available through Prometheus Adapter, and external metrics such as SQS queue depth suit worker pools. When several metrics are configured, the HPA computes a target for each and uses the largest, so any one signal under pressure triggers a scale up.

Why does my HPA keep scaling up and down repeatedly?

Usually the scale-down stabilization window is too short, so a brief dip removes capacity that is immediately needed again. The default of 300 seconds makes the HPA take the highest recommendation from the last five minutes, which damps exactly this. The other common cause is a conflict: if the Deployment manifest still specifies replicas and something reapplies it, the HPA and your deployment tooling will fight over the same field on every apply.

Do I need Cluster Autoscaler as well as the HPA?

Almost certainly. The HPA adds Pods; it does not add machines. If the cluster has no spare capacity the new Pods simply sit in Pending and no extra traffic is served. Cluster Autoscaler adjusts node group sizes in response, and Karpenter does the same job by provisioning right-sized instances directly and generally reacts faster. Set maxReplicas to something your cluster and budget can genuinely absorb, treating it as a safety limit rather than an aspiration.

Can I use HPA and Vertical Pod Autoscaler together?

Not on the same metric. VPA adjusts CPU and memory requests while the HPA scales replicas based on utilisation of those requests, so both react to each other and the result oscillates. The workable split is VPA managing memory and the HPA scaling on CPU or a custom metric, which keeps them out of each other's way. VPA in recommendation-only mode is always safe to run alongside an HPA, since it reports suggested values without changing anything.