Kubernetes incident debugging

At a glance

Symptom	Failing layer	First command	Where the cause usually is
Pod stuck Pending	Scheduler	kubectl describe pod	Resource requests, node capacity, taints, unbound volume claims
ImagePullBackOff	Kubelet and registry	kubectl describe pod	Wrong tag, missing pull secret, registry unreachable
CrashLoopBackOff	Container process	kubectl logs --previous	The application failing during startup, often config or a missing dependency
OOMKilled, exit code 137	Kernel and limits	kubectl describe pod	Memory limit below real usage, or a leak reaching it
Running but never Ready	Probe configuration	kubectl describe pod	Readiness probe path, port, or timing wrong for the application
502 or 503 at the ingress	Service and endpoints	kubectl get endpoints	No ready backends behind the Service, or a port mismatch

How it works

The single most useful move in a Kubernetes incident is not a command. It is deciding which layer failed, because the same visible symptom, users cannot reach the application, is produced by at least six distinct failures with nothing in common.

The layers run in order and each one gates the next. The scheduler has to place the pod. The kubelet has to pull the image. The container process has to start and stay running. The readiness probe has to pass. The Service has to have ready endpoints behind it. The ingress has to route to that Service. A failure at any layer produces the same user-visible outcome and a completely different fix.

This is why reaching for kubectl logs first is the most common wasted move. Logs only exist if the container process actually started, so on a Pending or ImagePullBackOff pod there is nothing to read, and the emptiness gets misread as an application problem. kubectl describe pod is the better opening move for anything below the process layer, because it prints the scheduler and kubelet events that explain why the pod is where it is.

Once the container has started at least once, kubectl logs --previous matters more than kubectl logs. After a crash the current container is a fresh instance, so the plain command shows a process that has not failed yet while the evidence sits in the previous one.

Two facts about events are worth knowing before an incident rather than during one. Events are namespaced, so kubectl get events in the wrong namespace looks reassuringly empty. And they expire: the API server keeps them for one hour by default, which means the events explaining the original failure are frequently gone by the time a long incident is being investigated.

That expiry is the structural limit of kubectl as a debugging surface. It shows current state well and keeps almost no history, so it can tell you a pod is CrashLoopBackOff and not that the deploy forty minutes ago changed the config map it reads at startup. Answering that means correlating cluster state with change history, which is the job NOFire AI does by tying a symptom back to the specific deploy or configuration event that caused it, with the dependency path attached.

How it is measured

Measure the layer identification, not the total.

Time to correct layer is the number that matters, because it is the part of a Kubernetes incident that varies most between responders. An engineer who recognises ImagePullBackOff as a registry problem is finished in a minute; one who starts reading application logs can lose twenty. Track it by asking, in the retrospective, which layer the first three commands were aimed at and whether that was the failing one.

First-hypothesis accuracy is the same discipline applied to the diagnosis rather than the tool, and it is the measurement automated root cause analysis is scored on. The version that matters here is narrower: how often was the first layer guessed correctly.

Two things are worth counting across incidents rather than within one. How many were caused by a change in the preceding hour, which tells you whether your real problem is deployment risk rather than debugging skill. And how many required information that had already expired, which is a direct measurement of whether your event retention and log aggregation are adequate.

Mean time to resolution is the wrong instrument at this level. It aggregates detection, layer identification, diagnosis and remediation into one number, and the whole point of the layer model is that those are separable.

Common misconceptions

That logs are the first place to look. They are the first place to look once you know the process started. Below that layer they are empty, and the emptiness is misleading rather than informative.

That CrashLoopBackOff is an error. It is a backoff state, not a cause. It says the container has exited repeatedly and the kubelet is now waiting longer between attempts. The cause is whatever made the process exit, which is in the previous container's logs.

That a Running pod is a working pod. Running means the container process exists. Ready means the readiness probe passes. A pod can sit Running and never Ready indefinitely, serving nothing, and the deployment will report it as unavailable while the pod looks alive.

That 503 means the application is down. It frequently means the Service has no ready endpoints, which is a routing and probe problem rather than an application one. The application may be running perfectly and simply not be in the endpoint list, which is why debugging 502 and 503 errors starts at the endpoints rather than the pods.

That raising the memory limit fixes OOMKilled. It stops the symptom and tells you nothing about whether usage is legitimate growth or a leak. A limit raised without that answer is a longer interval before the same incident.

That kubectl is a record. It is a view of current state. Events expire, replaced pods take their logs with them, and nothing in the cluster tells you what changed. That gap is where most of the time in a long Kubernetes incident actually goes.