Kubernetes HPA Generator
Build a HorizontalPodAutoscaler on autoscaling/v2, with the CPU target, the replica range and both stabilization windows written out. It tells you what the percentage is a percentage of, and says so plainly when the target pods have no CPU request and the autoscaler therefore cannot work at all.
hpa.yaml
updates as you typeCommon mistakes
These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.
Not setting resource requests
The HPA computes utilisation as a percentage of the request. With no request there is no denominator, so it reports unknown and never scales.
Instead:Set CPU and memory requests on every container the HPA targets.
Expecting the HPA to scale on memory usefully
Most applications do not release memory back, so memory utilisation ratchets up and never comes down, which scales up and never down.
Instead:Scale on CPU or a custom metric. Memory is rarely a good signal.
Leaving the default stabilisation window
Scale-down waits five minutes by default, which reads as the HPA being broken when traffic drops.
Instead:Tune behavior.scaleDown if faster is genuinely wanted, knowing it risks flapping.
What a HorizontalPodAutoscaler actually measures
An HPA reads an average metric across the ready pods of one workload, divides it by the target you set, and writes a new replica count to that workload's scale subresource. Everything that goes wrong with one comes from a detail in that sentence: what the percentage is a percentage of, where the metric comes from, which pods count towards the average, and who else is writing to the same field.
Utilization is a percentage of the request, not the limit
averageUtilization is measured against resources.requests.cpu on the pod, so a target of 70% on a container requesting 200m means the HPA holds the average near 140m. It has no relationship to the CPU limit and none to the node. The consequence people hit is the one with no error message: a pod with no CPU request gives the controller nothing to divide by, so the metric is unknown, and an HPA with an unknown metric does not scale in either direction. It reports this and carries on, indefinitely.
requests.cpu: 200m, target 70%
-> HPA acts on ~140m average per pod
-> a pod using 200m reads as 100%
-> a pod using 400m reads as 200%
requests.cpu absent:
$ kubectl get hpa
NAME TARGETS MINPODS MAXPODS REPLICAS
api-hpa <unknown>/70% 2 10 2
^ permanent, not "still starting"
Raising the request lowers the percentage for the
same real load. Requests and targets are tuned
together or not at all.The formula, and the 10% band where nothing happens
The controller runs every 15 seconds, computes the average over ready pods, and applies one line of arithmetic. It then ignores its own answer if the ratio is within 10% of 1.0, which is the cluster-wide tolerance on kube-controller-manager. So a target that is slightly overshot is not a problem to tune away: it is inside the deadband on purpose, because scaling on a 3% overshoot would mean scaling constantly.
desiredReplicas =
ceil(currentReplicas * (currentMetric / targetMetric))
4 replicas, average 105%, target 70%
ceil(4 * 1.5) = 6 replicas
10 replicas, average 35%, target 70%
ceil(10 * 0.5) = 5 replicas
tolerance, target 70%:
63% .. 77% -> ratio within 10% of 1.0, no action
Pods that are not Ready are excluded from the
average. Pods with no metric yet count as 0% when
scaling up and 100% when scaling down, so a missing
reading never causes a scale event by itself.Scale up is immediate, scale down waits five minutes
behavior.scaleUp.stabilizationWindowSeconds defaults to 0 and behavior.scaleDown.stabilizationWindowSeconds defaults to 300. The asymmetry is deliberate. Being under-provisioned costs requests now, so the controller adds pods on the first reading that asks for it. Removing pods is the dangerous direction, so it takes the highest recommendation from the last five minutes, which means load has to stay down for the whole window before a single pod goes. Shorten the window and the autoscaler follows every dip in traffic, removing pods it wants back a minute later, each cycle paying a pod start and, on a JVM, a cold JIT.
behavior:
scaleUp:
stabilizationWindowSeconds: 0 # default
scaleDown:
stabilizationWindowSeconds: 300 # default
scale down, window 300s:
t+0s recommendation 4
t+60s recommendation 3
t+120s recommendation 6 <- highest in window
t+180s recommendation 3
actual replicas: 6, until the spike ages out
Leave the pair out entirely and you get exactly
these numbers. They are written into the output so
the asymmetry is visible rather than assumed.An HPA and spec.replicas fight over the same field
Both write the replica count of the workload. kubectl apply sets it from the manifest, then the controller sets it back within about 15 seconds, and the pods start and stop for as long as anything keeps applying. With a GitOps controller reconciling continuously that is permanent, and it shows up as a workload that will not settle rather than as an error anywhere. The fix is to remove replicas from the Deployment, which is what our Kubernetes YAML generator does whenever it emits an HPA.
# Deployment: no replicas field at all
spec:
selector:
matchLabels: { app: api }
template: ...
# HPA owns the count from here on
spec:
scaleTargetRef:
kind: Deployment
name: api
minReplicas: 2
maxReplicas: 10
# The HPA also cannot scale a DaemonSet: it writes
# to the scale subresource and a DaemonSet has none.Multiple metrics take the highest, and memory rarely helps
Add a second metric and the HPA does not average or combine them. It computes a desired replica count from each metric independently and uses the largest, so a second metric can only ever raise the count. That makes memory a poor second signal for most JVM and Go services: neither returns freed heap to the operating system by default, so the working set rises to its high-water mark and stays there. The HPA scales up on it and then never scales down, because the number never falls. Queue depth or requests per second through an external metric describes load; memory describes the allocator.
cpu -> ceil(4 * (90/70)) = 6
memory -> ceil(4 * (60/80)) = 3
result: 6
Two more scoping details:
Resource averages over the SUM of every container
request in the pod, so an idle sidecar joins the
denominator and drags the percentage down. Use
ContainerResource (1.27+, stable in 1.30) to target
one container instead.
minReplicas: 0 needs the alpha HPAScaleToZero gate
and an Object or External metric. On CPU it cannot
work: ceil(0 * anything) is 0, so there is no way
back up.It is not a scheduler, and metrics-server is not installed
The HPA writes a number and stops. If there is no room in the cluster, the extra pods sit Pending and the HPA still reports the scale as achieved, so a saturated node pool looks like a working autoscaler from the HPA's side. Finding capacity is the cluster autoscaler's or Karpenter's job. The other half of this is the supply of metrics: metrics-server is a separate component that GKE and AKS ship and that kubeadm and EKS clusters do not, and without it there are no resource metrics at all.
$ kubectl top pods
error: Metrics API not available
-> metrics-server is missing. Every HPA in the
cluster reads <unknown>. Fix this first.
$ kubectl describe hpa api-hpa
ScalingActive True ValidMetricFound
ScalingLimited True TooManyReplicas
-> the HPA wanted more than maxReplicas
$ kubectl get pods
api-7d9f-x 0/1 Pending
-> the HPA did its part; there is no node with
room for the pod it asked for