Faster root cause analysis for cascading failures within a Kubernetes cluster

At a glance

Symptom and event reason	Failing layer	First command	kube-state-metrics series	Where the cause usually is
Pod stuck Pending, event FailedScheduling	Scheduler	kubectl describe pod	kube_pod_status_unschedulable	Resource requests, node capacity, taints, unbound volume claims
ImagePullBackOff, events Failed and BackOff	Kubelet and registry	kubectl describe pod	kube_pod_container_status_waiting_reason	Wrong tag, missing pull secret, registry unreachable
CrashLoopBackOff, event BackOff	Container process	kubectl logs --previous	kube_pod_container_status_restarts_total	The application failing during startup, often config or a missing dependency
OOMKilled, exit code 137	Kernel and limits	kubectl describe pod	kube_pod_container_status_last_terminated_reason	Memory limit below real usage, or a leak reaching it
Pod Evicted, node event NodeHasDiskPressure	Kubelet and node resources	kubectl describe node	kube_node_status_condition	Disk, memory or process ID pressure on the node
Running but never Ready, event Unhealthy	Probe configuration	kubectl describe pod	kube_pod_container_status_ready	Readiness probe path, port, or timing wrong for the application
502 or 503 at the ingress	Service and endpoints	kubectl get endpoints	kube_deployment_status_replicas_available	No ready backends behind the Service, or a port mismatch

The kube-state-metrics series record the same state in Prometheus, where it outlasts the event.

How it works

The most useful move in a Kubernetes incident is deciding which layer failed, before running any command. The same visible symptom, users cannot reach the application, is produced by at least six distinct failures with nothing in common.

The layers run in order and each one gates the next: scheduler, image pull, container process, node resources, readiness probe, Service endpoints, ingress. A failure at any layer produces the same user-visible outcome and a completely different fix.

This is why reaching for kubectl logs first is the most common wasted move. Logs only exist if the container process started, so on a Pending or ImagePullBackOff pod the empty output gets misread as an application problem. kubectl describe pod prints the scheduler and kubelet events instead. After a crash, kubectl logs --previous reads the instance that failed.

Events are namespaced, so kubectl get events in the wrong namespace looks reassuringly empty. kubectl get events -A --field-selector type=Warning lists every warning in the cluster. The API server also deletes events after one hour by default. kube-state-metrics exports object state as Prometheus series, so a restart count or a node condition is still there after its event is gone.

CrashLoopBackOff, eviction and FailedScheduling

CrashLoopBackOff means the container keeps exiting, and the kubelet restarts it with a growing delay capped at five minutes. If the image has no shell, kubectl debug -it checkout-7d9f --image=busybox --target=app attaches an ephemeral debug container, stable since Kubernetes 1.25. If the process dies on startup, kubectl debug checkout-7d9f -it --copy-to=checkout-debug --container=app -- sh starts a copy of the pod with a shell in place of the failing command.

Node-pressure eviction is the kubelet stopping pods to reclaim memory, disk or process IDs. It sets a node condition such as DiskPressure, records NodeHasDiskPressure, and evicts first the pods that use more than they request. The evicted pod stays Failed with reason Evicted, and its controller creates a replacement pod. kubectl describe node shows the conditions.

FailedScheduling is the scheduler's event for a pod it cannot place. The message counts nodes per reason, for example 0/6 nodes are available: 3 Insufficient cpu. Read the reasons before you add nodes. More capacity fixes Insufficient cpu, and it does not bind an unbound volume claim. After an eviction wave, replacements that find no room on the remaining nodes land here. One full disk then becomes pods stuck Pending across a deployment.

Accelerating root cause analysis

In a cascading failure, dozens of pods report symptoms at once, and reading them one by one mostly finds victims. Start from the change instead. Changes in the window are few: a handful of deploys, config map edits, HPA adjustments and node events. Discard every symptom that sits downstream of another symptom, and what remains is the earliest failing service and the change on it.

Kubernetes tools cover parts of this. K8sGPT, an open-source CLI, runs analyzers over cluster objects and can ask an LLM to explain each finding. Robusta attaches pod logs, events and recent resource changes to Prometheus alerts. Komodor keeps a timeline of changes across Kubernetes resources.

NOFire AI is a production ops platform for engineers and AI agents. It starts when the alert fires and narrows the alerting services to the one upstream of the rest. It then returns the deploy, config or code change on that service, with every claim linked to evidence.

Mapping service dependencies and blast radius

Mapping dependencies between microservices shows the blast radius of a production incident, meaning the services a failure can still reach. The method has four steps:

1. Build the map from observed traffic. eBPF tools such as Pixie and Groundcover record service-to-service calls without code changes. Manifests and service catalogs miss calls that nobody declared.
2. Date every edge and every change, including config maps, image tags, replica counts and node events. You can then read the cluster as it stood when the failure began.
3. Join changes to alerts. The candidate cause is a change on a service with a dependency path to the first alert.
4. Walk forward from that change. The services it still affects are its blast radius, and the fix has to cover all of them.

Remediation within policy guardrails

If a change caused the cascade, revert it, for example with kubectl rollout undo deployment/checkout. Restarting victim pods clears symptoms that return while the change is live. Review the runbook after each incident that used it, because runbooks go stale as services change.

An automated remediation step can widen an incident. Before it runs, compute its predicted blast radius, check it against a policy, and record who approved it. In NOFire AI, reads go through, writes wait for a person, and every action is signed into the tools you already audit. An agent action whose predicted blast radius exceeds the runtime policy bound is refused or held for a person.

How it is measured

Time to correct layer is the number that matters, because it varies most between responders. An engineer who recognises ImagePullBackOff as a registry problem is finished in a minute. One who starts reading application logs can lose twenty. In the retrospective, ask which layer the first three commands targeted and whether that was the failing one.

First-hypothesis accuracy applies the same discipline to the diagnosis, and it is the measurement automated root cause analysis is scored on.

Two things are worth counting across incidents. How many followed a change in the preceding hour, which tells you whether your real problem is deployment risk. And how many needed information that had already expired, which measures whether your event retention and log aggregation are adequate.

Common misconceptions

That CrashLoopBackOff is an error. It is a backoff state, not a cause. The container has exited repeatedly and the kubelet now waits longer between attempts. The cause is whatever made the process exit, which is in the previous container's logs.

That an Evicted pod ran out of its own memory. OOMKilled is the kernel stopping one container at its memory limit, and the pod stays on its node. Eviction is the kubelet removing whole pods to protect the node, for example when the node disk fills. The fix is headroom on the node, or requests that match real usage.

That a Running pod is a working pod. Running means the container process exists. Ready means the readiness probe passes. A pod can sit Running and never Ready, serving nothing, while the deployment reports it as unavailable.

That 503 means the application is down. It frequently means the Service has no ready endpoints, which is a routing and probe problem. The application can be running and missing from the endpoint list, which is why debugging 502 and 503 errors starts at the endpoints.

That raising the memory limit fixes OOMKilled. It stops the symptom and tells you nothing about whether usage is legitimate growth or a leak. A leak reaches the new limit too.