Kubernetes CrashLoopBackOff Guide

CrashLoopBackOff tells you the kubelet is waiting before the next restart, and nothing about why. This is where to look instead, in the order that finds it fastest. Filter by exit code, status or symptom.

18 entries. Use your browser's find, Ctrl+F or Cmd+F, to jump to one.

What the status actually means

CrashLoopBackOff is not a failure mode. It is the kubelet saying it has restarted this container several times and is now waiting before the next attempt.

CrashLoopBackOff

The container keeps exiting, so restarts are being delayed

The kubelet restarted the container, it exited again, and this has happened enough times that the kubelet is now backing off. The status says nothing about why. It is a timer, not a diagnosis, and every minute spent searching for the meaning of CrashLoopBackOff is a minute not spent reading the previous container's logs.

What to do: Get the exit code and the previous logs. Those two answer it almost every time.

kubectl logs <pod> --previous
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

The back-off schedule

10s, 20s, 40s, 80s, 160s, then 5 minutes forever

The delay doubles after each failure and is capped at five minutes. The part that surprises people is the reset condition: the timer only goes back to ten seconds after the container has run successfully for ten continuous minutes. A container that starts, works for eight minutes and then dies never resets, so it stays at the five-minute ceiling indefinitely, and a fix deployed to it appears to do nothing for five minutes.

What to do: After deploying a fix, delete the pod rather than waiting. A new pod starts with a fresh back-off timer.

kubectl delete pod <pod>   # the Deployment recreates it immediately

RESTARTS

Reading the restart count

The number in kubectl get pod is the total for the pod's lifetime, and the time in brackets is how long since the last restart. A count that is high but static means the container has stabilised. A count climbing every few minutes is a live crash loop.

kubectl get pod -w

NAME       READY   STATUS             RESTARTS        AGE
api-7d9f   0/1     CrashLoopBackOff   6 (2m11s ago)   14m

The causes, roughly in order of how often they are it

All of these produce the same CrashLoopBackOff status. The exit code and the events tell them apart.

Application error on startup

Usually exit code 1

The process started and decided it could not continue: a missing environment variable, an unreachable database, a config file that does not parse. This is the most common cause by a wide margin, and the reason is printed in the logs of the instance that died.

What to do: Read the previous logs. The answer is almost always in the first ten lines.

kubectl logs <pod> --previous --tail=50

OOMKilled

Exit code 137 with reason OOMKilled

The container exceeded its memory limit during startup. A JVM is the classic case: older JVMs read the host's total memory rather than the cgroup limit and size a heap far larger than the container is permitted, so the process is killed before it finishes booting.

What to do: Raise limits.memory, or set the heap explicitly against the limit. Note that a memory-backed emptyDir, including /dev/shm, counts towards the limit.

kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

Exit code 0

It succeeded, and that is the problem

A Deployment's pod template must use restartPolicy: Always, so a container that completes successfully is started again immediately. Do that a few times and the pod reports CrashLoopBackOff without anything ever having failed. The usual cause is a process that daemonises itself, or a workload that is genuinely a batch job.

What to do: If it is meant to finish, make it a Job. If it is meant to stay up, stop the process forking into the background.

Liveness probe failing

The container is killed, not crashing

The application is fine and the liveness probe says otherwise, so the kubelet restarts it, repeatedly. The tell is in the events rather than the logs: the logs look like a clean startup followed by a clean shutdown, over and over. A liveness probe with too short an initialDelaySeconds on a slow-starting application produces exactly this, and it can never recover on its own.

What to do: Add a startupProbe so liveness does not begin until the application is up. Increase initialDelaySeconds only as a stopgap: it delays every later check too.

kubectl describe pod <pod> | grep -i "liveness\|unhealthy"

Missing ConfigMap or Secret

Often CreateContainerConfigError, not CrashLoopBackOff

A referenced ConfigMap or Secret does not exist, or the key inside it does not. If it is mounted or referenced with configMapKeyRef the pod never starts and the status is CreateContainerConfigError. With envFrom and a key that is not a valid shell identifier, the key is skipped silently and the application starts without it, which then fails as an ordinary application error.

What to do: Check the events, then check the referenced object exists in the same namespace.

kubectl describe pod <pod> | sed -n '/Events:/,$p'

Read-only root filesystem

The container dies on its first temp file

readOnlyRootFilesystem: true without a writable /tmp kills most images the moment they try to write anything, which for many runtimes is during startup. The error is a permission denied on a path nobody thought about.

What to do: Mount an emptyDir at /tmp, and at any other path the process writes to.

Wrong command or entrypoint

Exit code 127 or 126

The command does not exist in the image, or cannot be executed. 127 is not found, 126 is found but not runnable. A shell script with Windows line endings produces 127 and is invisible in every diff, because the interpreter line reads as /bin/sh followed by a carriage return.

What to do: Check the binary exists and the script has Unix line endings. Also remember that in Kubernetes, command replaces the image's ENTRYPOINT and args replaces its CMD.

States people confuse with it

These look similar in kubectl get pod and mean something entirely different. None of them are a crashing container.

ImagePullBackOff

The image could not be pulled

Nothing has run at all. The image name is wrong, the tag does not exist, or the registry needs credentials the node does not have. On a private registry this is nearly always a missing or wrong imagePullSecret, and the message is the same as for a genuinely missing image.

What to do: Check the image reference, then check the pull secret exists in this namespace. A secret in another namespace does not apply.

kubectl describe pod <pod> | grep -A3 Failed

CreateContainerConfigError

A referenced object is missing

The container could not be configured, usually because a ConfigMap, Secret or key it references does not exist. It is also what runAsNonRoot produces when the image's user cannot be resolved to a numeric uid.

What to do: The events name the missing object directly.

Init:CrashLoopBackOff

An init container is the one crashing

Init containers run to completion, in order, before any application container starts. One that fails blocks everything after it forever, and kubectl logs without -c shows the application container, which has never run and has no output.

What to do: Name the init container explicitly when reading logs.

kubectl logs <pod> -c <init-container-name> --previous

Pending

Nothing has been scheduled

The pod has not been placed on a node, so no container has started. Insufficient CPU or memory, a nodeSelector matching nothing, an unsatisfiable topology constraint, or a PVC that cannot bind. The reason is in the events, and it is specific.

kubectl describe pod <pod> | grep -A10 Events

The order to work through it

1. Previous logs

Not the current ones

The running container has usually printed nothing yet. Everything that explains the crash belongs to the instance that already exited.

kubectl logs <pod> --previous --tail=50

2. The exit code and reason

Which narrows it to a category immediately

137 with OOMKilled is memory. 137 with Error is an ignored SIGTERM. 1 is the application. 127 or 126 means it never ran. 0 means it succeeded and should not have been a Deployment.

kubectl get pod <pod> \
  -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

3. Events

For everything that happened outside the container

Probe failures, image pulls, scheduling, evictions and missing references are all here and none of them appear in the logs. Events are garbage collected after about an hour, so a pod that has been failing all night may have none left.

kubectl describe pod <pod> | sed -n '/Events:/,$p'

4. Run it by hand

When the logs say nothing

Start the same image with a shell instead of its entrypoint and look at the filesystem, the environment and the config it was given. This is the fastest way to find a missing file or a permission problem.

kubectl run debug --rm -it --restart=Never \
  --image=<same-image> --command -- sh

# or attach a debug container to the real pod:
kubectl debug -it <pod> --image=busybox --target=<container>

Common mistakes

These are the ones that fail silently. The config is accepted, nothing raises an error, and the consequence arrives later.

  1. Treating CrashLoopBackOff as a cause

    It is a back-off timer stating that the container keeps exiting. It says nothing about why, and the exit code and previous logs do.

    Instead:kubectl logs --previous, then the exit code. The status itself carries no diagnostic information.

  2. Reading only the current logs

    The current container has just started and has not failed yet. The useful output is from the instance that died.

    Instead:Always use --previous for a crash looping pod.

  3. Increasing the restart budget to make it stop

    There is no such setting for a Deployment, and the back-off exists to stop a broken pod consuming the node. Making restarts faster makes the node worse.

    Instead:Fix the exit. The back-off is a symptom of that, not a policy problem.

CrashLoopBackOff is a timer, not a diagnosis

It means the kubelet has restarted this container several times and is now waiting longer between attempts. That is the whole of it. The status carries no information about why, which is why searching for the status itself leads nowhere useful.

The back-off doubles, caps at five minutes, and rarely resets

Ten seconds, then twenty, forty, eighty, a hundred and sixty, then five minutes from there on. The reset condition is the part that catches people: the timer only returns to ten seconds after the container has run successfully for ten continuous minutes. A container that works for eight minutes and then dies never resets, so it sits at the five-minute ceiling indefinitely.

restart 1   after 10s
restart 2   after 20s
restart 3   after 40s
restart 4   after 80s
restart 5   after 160s
restart 6+  after 300s, forever

# deployed a fix? do not wait five minutes:
kubectl delete pod <pod>   # fresh pod, fresh timer

The logs you want belong to a container that is already gone

kubectl logs shows the container running now, which on a crash loop has usually printed nothing yet. The output that explains the crash belongs to the instance that exited, and it is one flag away. Once the pod itself is replaced, neither works and you are down to events, which are garbage collected after about an hour.

kubectl logs api-7d9f              often empty
kubectl logs api-7d9f --previous   the one that died

# multi-container pod? name it, or you get the wrong one:
kubectl logs api-7d9f -c sidecar --previous

# init container? it is a different flag again:
kubectl logs api-7d9f -c wait-for-db --previous

Two exit codes cover most of it, and one is ambiguous

Exit 1 is the application choosing to stop, so the answer is in its logs. Exit 137 is SIGKILL and has two unrelated causes: the memory limit, or a container that ignored SIGTERM until the grace period ran out. Only lastState.terminated.reason separates them, and raising the memory limit does nothing for the second.

kubectl get pod <pod> -o jsonpath=\
  '{.status.containerStatuses[0].lastState.terminated}'

{"exitCode":1,"reason":"Error"}          read the logs
{"exitCode":137,"reason":"OOMKilled"}    memory limit
{"exitCode":137,"reason":"Error"}        ignored SIGTERM
{"exitCode":0,"reason":"Completed"}      it succeeded

A liveness probe can cause a crash loop the logs will not explain

If the probe is failing, the application is being killed rather than crashing. The logs show a clean startup and a clean shutdown, over and over, with nothing wrong in them. The evidence is in the events instead. A liveness probe whose initialDelaySeconds is shorter than the application's start time produces this and can never recover on its own, because the container is killed before it ever finishes booting.

kubectl describe pod <pod> | grep -i unhealthy

Warning  Unhealthy  Liveness probe failed: Get
"http://10.1.2.3:8080/health": context deadline exceeded

# the fix is a startupProbe, not a longer initialDelay:
startupProbe:
  httpGet: {path: /health, port: 8080}
  failureThreshold: 30
  periodSeconds: 10        # 300s to boot, then liveness
                           # takes over at its own pace

Exit code 0 is a crash loop too

A Deployment's pod template is required to use restartPolicy: Always, so a container that finishes successfully is started again immediately. After a few rounds the pod reports CrashLoopBackOff having never failed at anything. The usual causes are a process that daemonises itself, and a batch workload that should have been a Job.

kubectl logs report-x --previous
Report generated successfully.

kubectl get pod
report-x   0/1   CrashLoopBackOff   5

# nothing is broken. It is a Job, not a Deployment.

More kubernetes tools

Kubernetes YAML Generator Answer a few questions, get a manifest Kubernetes YAML Validator Is this YAML valid, and where is it wrong? Kubernetes Rollout & Probe Timing Restart loops, capacity dips and stuck drains Kubernetes Service Checker Will this Service have any endpoints? Kubernetes Network Policy Viewer What can reach what, and what just broke Kubernetes Manifest Diff What actually changed Kubernetes Resource Calculator What these manifests actually reserve Kubernetes RBAC Viewer Who can actually do what Kubernetes Deployment Generator See the capacity dip before you deploy Kubernetes Service Generator port, targetPort, and which one you meant Kubernetes StatefulSet Generator Stable names, and volumes that outlive you Kubernetes QoS Class Calculator And where you sit in the eviction order Kubernetes Node Capacity Planner Allocatable is not capacity Kubernetes Max Pods per Node Calculator The ENI limit, not the 110 default Kubernetes Label Selector Tester Which key failed, not just that one did Kubernetes Taint & Toleration Tester Tolerations permit, they do not attract Kubernetes Resource Name Validator RFC 1123 subdomain, label, or 1035 Kubernetes Label & Annotation Validator Why your image tag is not a legal label Kubernetes TLS Certificate Decoder When it expires, and what it covers Kubernetes ServiceAccount Token Decoder Bound or legacy, and when it dies Kubernetes imagePullSecret Generator The registry URL nobody gets right Kubernetes cert-manager Certificates Issuer scope, and the rate limit that costs a week Kubernetes Flux HelmRelease Generator With the source it cannot work without Kubernetes Helm Chart Generator Chart.yaml, values, and working templates Kubernetes Argo CD Application Generator The defaults that are off, turned on Kubernetes Ingress Generator pathType, and the TLS secret that fails quietly Kubernetes HPA Generator Why it says unknown and never scales Kubernetes SecurityContext Generator Pod level, container level, and which wins Kubernetes Probe Generator How long it gets to start before liveness kills it Kubernetes PVC Generator Access modes your storage can actually do Kubernetes ResourceQuota Generator And the LimitRange that stops it breaking deploys Kubernetes CronJob Generator The fields that decide whether it runs Kubernetes NetworkPolicy Generator With the DNS rule already in it Kubernetes RBAC Generator And what it actually grants Kubernetes PodDisruptionBudget Generator Can this budget ever be satisfied? Kubernetes API Deprecation Checker What breaks on the next upgrade Kubernetes Kubeconfig Viewer Read it without handing it to anyone Kubernetes Quantity Parser What 512Mi, 100m and 1e3 actually mean Kubernetes Multi-Document YAML Splitter One file per object, in apply order Kubernetes .env to ConfigMap & Secret Split the credentials out on the way Kubernetes Secret Decoder See what is actually in there Kubernetes Ingress to Gateway API Which annotations silently stop working Kubernetes Manifest Visualizer What points at what, and what points at nothing Kubernetes Patch Tester Which --type, and what it deletes Kubernetes kubectl JSONPath Tester kubectl's dialect, not generic JSONPath Kubernetes Node Affinity Tester Required excludes, preferred only scores Kubernetes Helm Values Diff A missing key is a third value Kubernetes Pod and Service CIDR Fixed at cluster creation Kubernetes Secret Encoder base64 is not encryption Kubernetes API Version Checker Will this apply on that cluster Kubernetes Go Template Tester Go truthiness is not JavaScript's Kubernetes Cluster Cost Estimator At your rates, not a price table Kubernetes Kustomize Build Transformers run in kustomize's order Kubernetes Helm Chart Validator version is SemVer, appVersion is not

Elsewhere on the site