Kubernetes Rollout & Probe Timing

Work out what your probe, rollout and shutdown settings actually do: how long a container has to start before it is killed, how much capacity a deploy removes, and whether a node drain can complete at all.

Paste below, or drop a file anywhere on this panel

Or drop a file anywhere on this panel. Nothing is uploaded: the analysis runs in this tab.

The answer appears here

Paste on the left and press Work it out. Nothing leaves this tab.

Examples

Real input you can load into the tool above. Each one shows a different thing going wrong, because that is what the tool is for.

A slow rollout

maxUnavailable zero with a long readiness delay, which makes twenty replicas take a while

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 20
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: app
          readinessProbe:
            initialDelaySeconds: 30
            periodSeconds: 10

A faster strategy

The same rollout with surge raised, and what it costs in peak capacity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 20
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
  template:
    spec:
      containers:
        - name: app

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Setting maxUnavailable to zero without surge

    With both at zero nothing can move and the rollout stalls. With surge one it proceeds one pod at a time.

    Instead:Allow either surge or unavailability, and expect the rollout to take replicas times readiness time.

  2. Forgetting the preStop hook

    Without one, a pod stops accepting traffic and is killed before endpoints propagate, dropping in-flight requests.

    Instead:Add a preStop sleep of a few seconds.

  3. Ignoring initialDelaySeconds in the estimate

    Every replica waits it out. Thirty seconds times twenty replicas at maxSurge one is ten minutes.

    Instead:Lower the delay and use a startup probe for slow-starting apps.

What it works out

None of these settings produce an error when they are wrong. They produce a restart loop, a capacity dip, or an upgrade that never finishes.

How long the container has to start

initialDelaySeconds plus failureThreshold times periodSeconds is the budget before the liveness probe kills it. An app needing 45 seconds under a probe that allows 20 restarts forever, and the logs show a clean startup every single time, which is what makes it so hard to diagnose.

maxSurge rounds up, maxUnavailable rounds down

That asymmetry is Kubernetes' own and it catches people. 25% of 3 replicas is 0.75, which becomes 1 extra pod of surge and 0 unavailable. So the setting you thought limited disruption often does nothing at all until the replica count grows.

PodDisruptionBudgets that block a drain forever

minAvailable equal to the replica count means no pod may ever be evicted. kubectl drain does not fail, it retries indefinitely, so a cluster upgrade or an autoscaler scale-down hangs with nothing explaining why. The arithmetic is resolved against the actual replica count, including percentages.

The requests dropped on every deploy

Endpoint removal and SIGTERM happen in parallel, not in order, so traffic keeps arriving at a pod that has already started shutting down until every kube-proxy notices. Without a preStop sleep those requests fail, on every rollout, and it looks like random flakiness.

The three clocks in a pod lifecycle

Startup, rollout and shutdown each have their own timers, spread across fields that do not sit near each other in the manifest. None of them produce an error when set wrongly. They produce restart loops, capacity dips and dropped connections instead.

Startup: the probe that should be doing the waiting

A liveness probe answers whether a running container is still healthy. It is not for waiting on a slow boot, but with no startupProbe present its clock starts when the container does, so it ends up doing that job badly. The startupProbe exists precisely to separate the two: give it a generous failureThreshold, and the liveness probe does not begin until startup has succeeded, so you can keep its threshold tight without punishing a slow start.

startupProbe:
  failureThreshold: 30
  periodSeconds: 5      allows 150s to boot

livenessProbe:
  failureThreshold: 3
  periodSeconds: 5      then kills after 15s of failures

without the startupProbe, that liveness probe kills at 15s
whatever the application is doing

Rollout: two numbers, rounded in opposite directions

maxSurge is how many extra pods may exist above the replica count, and it rounds up. maxUnavailable is how many may be missing, and it rounds down. Both default to 25%. The rounding is what makes small replica counts behave unexpectedly, and it is why maxSurge 1 with maxUnavailable 0 is the usual answer for anything that matters: it costs one extra pod during the rollout and loses no capacity at all.

3 replicas, both at 25%

  maxSurge        0.75 -> rounds UP   -> 1
  maxUnavailable  0.75 -> rounds DOWN -> 0

  between 3 and 4 pods exist during the rollout

Disruption budgets protect drains, not rollouts

A rollout is already governed by maxUnavailable. A PodDisruptionBudget governs voluntary eviction, which is node drains, cluster upgrades and autoscaler scale-downs. They are separate mechanisms and a workload wants both. The failure mode is specific: set minAvailable to the replica count and the eviction API refuses every request, drain retries in a loop, and the upgrade never finishes.

Shutdown is not ordered, which is the whole problem

When a pod is deleted, Kubernetes sends SIGTERM and removes the pod from the Service endpoints at the same time. Endpoint removal then has to propagate to every node's kube-proxy, which takes a moment. During that moment traffic is still being sent to a pod that has begun shutting down. A preStop hook that sleeps for a few seconds fixes it, because the container keeps serving normally while the sleep runs and the endpoint removal catches up.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "5"]

terminationGracePeriodSeconds: 45   covers the sleep AND the drain

More kubernetes tools

Kubernetes YAML Generator Answer a few questions, get a manifest Kubernetes YAML Validator Is this YAML valid, and where is it wrong? Kubernetes Service Checker Will this Service have any endpoints? Kubernetes Network Policy Viewer What can reach what, and what just broke Kubernetes Manifest Diff What actually changed Kubernetes Resource Calculator What these manifests actually reserve Kubernetes RBAC Viewer Who can actually do what Kubernetes Deployment Generator See the capacity dip before you deploy Kubernetes Service Generator port, targetPort, and which one you meant Kubernetes StatefulSet Generator Stable names, and volumes that outlive you Kubernetes QoS Class Calculator And where you sit in the eviction order Kubernetes Node Capacity Planner Allocatable is not capacity Kubernetes Max Pods per Node Calculator The ENI limit, not the 110 default Kubernetes Label Selector Tester Which key failed, not just that one did Kubernetes Taint & Toleration Tester Tolerations permit, they do not attract Kubernetes Resource Name Validator RFC 1123 subdomain, label, or 1035 Kubernetes Label & Annotation Validator Why your image tag is not a legal label Kubernetes CrashLoopBackOff Guide The status is a timer, not a cause Kubernetes TLS Certificate Decoder When it expires, and what it covers Kubernetes ServiceAccount Token Decoder Bound or legacy, and when it dies Kubernetes imagePullSecret Generator The registry URL nobody gets right Kubernetes cert-manager Certificates Issuer scope, and the rate limit that costs a week Kubernetes Flux HelmRelease Generator With the source it cannot work without Kubernetes Helm Chart Generator Chart.yaml, values, and working templates Kubernetes Argo CD Application Generator The defaults that are off, turned on Kubernetes Ingress Generator pathType, and the TLS secret that fails quietly Kubernetes HPA Generator Why it says unknown and never scales Kubernetes SecurityContext Generator Pod level, container level, and which wins Kubernetes Probe Generator How long it gets to start before liveness kills it Kubernetes PVC Generator Access modes your storage can actually do Kubernetes ResourceQuota Generator And the LimitRange that stops it breaking deploys Kubernetes CronJob Generator The fields that decide whether it runs Kubernetes NetworkPolicy Generator With the DNS rule already in it Kubernetes RBAC Generator And what it actually grants Kubernetes PodDisruptionBudget Generator Can this budget ever be satisfied? Kubernetes API Deprecation Checker What breaks on the next upgrade Kubernetes Kubeconfig Viewer Read it without handing it to anyone Kubernetes Quantity Parser What 512Mi, 100m and 1e3 actually mean Kubernetes Multi-Document YAML Splitter One file per object, in apply order Kubernetes .env to ConfigMap & Secret Split the credentials out on the way Kubernetes Secret Decoder See what is actually in there Kubernetes Ingress to Gateway API Which annotations silently stop working Kubernetes Manifest Visualizer What points at what, and what points at nothing Kubernetes Patch Tester Which --type, and what it deletes Kubernetes kubectl JSONPath Tester kubectl's dialect, not generic JSONPath Kubernetes Node Affinity Tester Required excludes, preferred only scores Kubernetes Helm Values Diff A missing key is a third value Kubernetes Pod and Service CIDR Fixed at cluster creation Kubernetes Secret Encoder base64 is not encryption Kubernetes API Version Checker Will this apply on that cluster Kubernetes Go Template Tester Go truthiness is not JavaScript's Kubernetes Cluster Cost Estimator At your rates, not a price table Kubernetes Kustomize Build Transformers run in kustomize's order Kubernetes Helm Chart Validator version is SemVer, appVersion is not

Elsewhere on the site