CrashLoopBackOff is not an error: reading the status Kubernetes actually gave you

CrashLoopBackOff is the most-searched Kubernetes pod status and one of the least informative, because it is not a diagnosis. It is a description of what the kubelet is doing at this instant: waiting, before it tries again.

The container exited. Kubernetes restarted it. It exited again. After a few rounds the kubelet stops restarting immediately and starts backing off, doubling the delay each time to a ceiling of five minutes. CrashLoopBackOff is the name of that waiting period, not of the failure that caused it.

So the status tells you a container is failing repeatedly. It tells you nothing about why, and the first instinct it produces returns nothing.

The two commands, in order

kubectl logs my-pod --previous
kubectl describe pod my-pod

That is the whole opening move. Everything below is about reading what they return.

Why --previous is the difference between logs and no logs

kubectl logs my-pod gives you the current container. During a backoff there either is no current container, or there is a freshly started one that has not failed yet. Either way the output that explains the crash belongs to the attempt that already ended.

--previous reads that one. It is the single most common missing flag in a runbook, usually because whoever wrote the runbook debugged interactively, typed it without thinking, and never noticed it was load-bearing.

If both are empty, the container is dying before it produces any output at all, and the answer is in the exit code instead.

Reading the exit code

In kubectl describe pod, look for Last State: Terminated and the Exit Code beside it.

Exit code	Almost always means	Where to look next
0	The process finished successfully	The command, not the cluster. See below
1	The application threw and exited	logs --previous
2	Shell misuse, or a missing binary in the entrypoint	The image, not the code
126	The command exists and is not executable	File permissions in the image
127	Command not found	The entrypoint path, or a missing shell in a distroless image
137	SIGKILL	OOM, or a liveness probe. See below
139	Segmentation fault	A native dependency, or an architecture mismatch
143	SIGTERM, and the process did not exit in the grace period	Signal handling

Two of these deserve their own sections because they mislead.

Exit code 0 is the confusing one

It looks like success. It is a failure of a specific and slightly philosophical kind: a Deployment expects a process that does not end.

A container whose command completes exits zero, and the kubelet restarts it, because a Deployment’s restartPolicy is Always and cannot be changed. The container is behaving perfectly and the pod is in CrashLoopBackOff.

The usual causes:

• A script that finishes rather than execing a long-running process.
• A server that daemonises and lets the foreground process return. Everything runs, briefly, in a container that then exits.
• A shell with nothing to keep it alive, often command: ["/bin/sh"] with no -c and no argument.
• A batch job deployed as a Deployment. It should be a Job or a CronJob, which have restartPolicy: OnFailure or Never and understand completion.

That last one is the interesting case, because nothing is broken at all. The workload finished. The object type is wrong.

137 means killed, and the two causes look nothing alike

If Reason: OOMKilled appears next to it, the container exceeded its memory limit. Raise the limit, or find the leak. Note that the limit is enforced against the whole cgroup, so a JVM or a Go runtime sized against the node’s memory rather than the container’s limit will do this reliably.

If Reason: OOMKilled is absent, something else sent SIGKILL, and on a CrashLoopBackOff the overwhelmingly likely candidate is a failing liveness probe.

The probe that kills a perfectly healthy container

This deserves its own section because the fix is counterintuitive and the symptom is distinctive.

A liveness probe with a short initialDelaySeconds starts checking before the application has finished booting. The check fails, the kubelet kills the container, the container restarts, and it fails the same check at the same point in its startup. The application is fine. The probe is executing it at the same moment every time, which is why the loop is so regular: the intervals are identical rather than varying the way a real crash would.

The wrong fix is a longer initialDelaySeconds. It works until startup gets slower, which it will: a bigger dataset to load, a cold cache, a slow dependency, a noisy neighbour on the node. Then somebody has to guess a new number. It is a magic constant that has to be re-tuned forever and will be wrong at the worst time.

The right fix is a startup probe. It runs first, holds the liveness probe off until it passes once, and can have a generous failureThreshold without weakening liveness afterwards:

startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30      # 30 x 10s = up to 5 minutes to start
  periodSeconds: 10
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  periodSeconds: 10         # only begins after startupProbe succeeds
  failureThreshold: 3

Startup time can now vary by five minutes without anybody touching a number, and a genuinely hung process is still killed within thirty seconds once it is up.

While you are there: liveness and readiness are different questions. Liveness asks “should this container be killed”, readiness asks “should traffic be sent”. Pointing both at the same endpoint means a dependency outage that fails readiness also fails liveness, and Kubernetes restarts every replica of a healthy service because a database is slow. Readiness should check the thing; liveness should check that the process is not wedged.

Trade-offs when the container is genuinely crashing

Once you know it is your process failing, the question is how to see it. Three options with real costs:

Approach	Cost	When to use it
logs --previous	Free	Always first. Fails only when the process dies before logging
Override the entrypoint to sleep 3600, then exec in	A deploy, and a pod that is not serving	The process dies before producing output, and you need the filesystem and env
kubectl debug -it pod/x --image=busybox --target=app	Requires ephemeral containers, and a shell image	Distroless images, where there is nothing to exec into

The middle one is the workhorse and is worth knowing precisely. Patch the command to something that will not exit, let the pod come up, then exec in and run the real entrypoint by hand. You get the actual error on your terminal rather than through the log pipeline, with the real environment variables, the real mounted config, and the real filesystem.

The failure it finds most often is not a code bug. It is a config or secret key that is absent, present but empty, or spelled differently from what the code reads.

The failures that are not in your container at all

Two statuses are routinely lumped in with CrashLoopBackOff and have nothing to do with your code:

ImagePullBackOff is the same backoff mechanism applied to pulling the image. The container never started, so there are no logs and no exit code. Check the tag exists, the registry is reachable from the node, and imagePullSecrets is set on the pod’s service account or the pod spec. Private registries in a different namespace from the one you tested in are the classic.

CreateContainerConfigError means a referenced ConfigMap or Secret does not exist, or a key inside it does not. The kubelet cannot construct the container, so there is nothing to crash. describe names the missing object, and it is worth reading the whole line: a missing key inside an existing ConfigMap looks very similar to a missing ConfigMap and is fixed differently.

What changed recently, and what it means for debugging

Two shifts worth knowing if your mental model is a few releases old:

Ephemeral containers are stable. kubectl debug attaching a container into a running pod is no longer a feature-gated experiment. This matters most for distroless and scratch images, where the traditional advice of “exec in and look around” was impossible because there was no shell. You can now attach one.

Sidecar containers are proper init containers with restartPolicy: Always. Before this, a sidecar that crashed could take down a pod in ways that were hard to attribute, and ordering between a sidecar and the main container was convention rather than guarantee. Now a sidecar starts before the main container and is not required to complete. If you are debugging a CrashLoopBackOff in a pod with a service mesh proxy, check which container is actually looping: describe lists them separately, and the answer is often the proxy rather than the app.

Fixing it in a running cluster

The pod is looping in production and you need it stable, not perfect:

1. Stop the churn if the backoff is hurting. Scale the Deployment to zero rather than leaving a crash loop hammering the API server, the image registry and your log pipeline. A pod in CrashLoopBackOff at a five minute interval is cheap; forty of them mid-rollout are not.

2. Check whether the rollout is stuck rather than complete. A Deployment with maxUnavailable: 0 will not proceed past a failing new ReplicaSet, so old pods are still serving. That is the system working: you have time. Confirm it with kubectl rollout status before treating this as an outage.

3. Roll back on the object, not by redeploying. kubectl rollout undo returns to the last working ReplicaSet immediately. Rebuilding and redeploying a fix under pressure is how a one-line config error becomes a two-hour incident.

4. Then debug the failing version with the entrypoint override above, on a pod that is not in the service.

When “just restart it” is the right answer

Worth stating plainly, because a post like this can imply every crash loop needs a root cause before you touch anything.

If the exit code is 137 with OOMKilled on a workload whose memory profile you already understand, raising the limit is the fix and there is nothing further to learn. If the container failed because a dependency was briefly unavailable at startup and the backoff has since recovered it, the loop was the system absorbing a transient failure exactly as designed.

The distinction is whether the loop terminated on its own. A pod that went CrashLoopBackOff and is now Running rode out a transient problem. A pod that has been looping for twenty minutes with a rising restart count has a cause that will not resolve itself, and the exit code is where to start.

The short version

• The status is a delay, not a diagnosis. It says nothing about why.
• kubectl logs --previous, then kubectl describe. In that order, every time.
• Exit 0 on a Deployment means the process completed. That is the bug.
• Exit 137 without OOMKilled is almost always a liveness probe.
• Fix probe-induced loops with a startup probe, not a longer initialDelaySeconds.
• ImagePullBackOff and CreateContainerConfigError are different problems wearing a similar hat.

The CrashLoopBackOff guide walks the exit codes, the probe interactions and the config failures in the order that eliminates the most possibilities per step, which is not the order they appear in describe output.