Debug Kafka consumer lag and poison pill incidents

Before you start

You need read access to the cluster with the Kafka CLI tools on the path, and if the topic uses Avro or Protobuf, read access to the Schema Registry REST API. Knowing which service runs the consumer group, and what that service calls synchronously while processing a record, saves the most time and is the thing least often written down.

Symptom	What it usually means	First check
Lag climbing, committed offset still advancing	Throughput. The consumer is processing but slower than production	Consumer CPU and per-record processing time
Lag climbing, committed offset frozen	A stall. Something is blocking on one record	Fetch the record at the stuck offset
One partition frozen, the rest healthy	A poison pill or a hot key on that partition	The stuck record and its schema ID
All partitions frozen after a consumer deploy	The consumer's own change	The deploy, then roll back
Group rebalancing repeatedly	Consumers being evicted, often from slow records	max.poll.interval.ms and session timeouts
Frozen offset, producer deployed recently	A schema or payload change upstream	Schema Registry version history for the topic

The steps

1. Decide whether this is a stall or a throughput problem. Describe the group, wait a minute, describe it again:

kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group orders-fraud-scorer

Compare CURRENT-OFFSET per partition between the two runs. If it advanced and LAG still grew, the consumer is too slow and this is a capacity investigation. If CURRENT-OFFSET did not move on any partition, the consumer is stalled and everything below applies. This one comparison is worth more than any dashboard, because a lag chart renders both cases as the same rising line.

2. Localise the stall to partitions. From the same output, note which partitions have a frozen offset. One partition stuck while the others advance almost always means a specific record. All partitions stuck at once points at the consumer process or something it depends on.

3. Fetch the record the consumer cannot get past. The stuck offset is CURRENT-OFFSET for that partition:

kafka-console-consumer.sh --bootstrap-server broker:9092 --topic orders.v1 \
  --partition 3 --offset 184223 --max-messages 1 --property print.key=true

If the payload is Avro or Protobuf through Schema Registry, the first byte is a magic byte and the next four are the schema ID. That ID is the next thing to check.

4. Check whether the schema moved. List the versions registered for the topic's value subject:

curl -s http://schema-registry:8081/subjects/orders.v1-value/versions
curl -s http://schema-registry:8081/subjects/orders.v1-value/versions/latest

If a new version appeared shortly before the stall and the consumer's reader schema is pinned to an older one, this is the case to look for. Under the default BACKWARD compatibility mode the registry approves a change like widening a field from int to long, correctly, because a new reader can still read old data. It says nothing about whether an old reader can read new data, and the consumer that is pinned to the old schema stalls without an error the registry ever saw.

5. Look past the consumer. If the record is fine and the schema has not changed, the consumer is probably blocked on something it calls while processing. Find what the consuming service depends on synchronously, then check whether any of those dependencies deployed or degraded at the onset of the stall. A fraud-scorer consuming orders.v1 and calling risk-engine per record will freeze its offset the moment risk-engine slows down, and nothing in Kafka has changed.

6. Line the stall up against the change timeline. Producer deploys, registry version bumps, consumer deploys, and deploys to anything the consumer calls. The stall began at a specific minute; one of those changes is usually within a few minutes of it. This is the same discipline as root cause analysis anywhere else: the first hypothesis should name a change, and a stall with no candidate change nearby is the signal to widen the search to the consumer's dependencies rather than to keep staring at the topic.

This walk, from topic to consumer to the consumer's dependencies to the change on one of them, is what NOFire AI does automatically when Kafka is connected. Clusters, topics and consumer groups become entities with dependency edges to the services that produce and consume them, so a stall reaches the same dependency map and change timeline as any other incident rather than stopping at the topic boundary.

Verify it worked

Whatever the fix, the test is the same: describe the group again and confirm CURRENT-OFFSET is advancing on the partitions that were stuck, and that LAG is falling rather than merely stable.

For a poison pill, confirm the consumer has moved past the offending offset specifically, since a restart can reprocess the same record and stall again on the next poll.

Confirm the group is stable. kafka-consumer-groups.sh --describe shows the member assignments, and a group that is still rebalancing has not recovered even if lag briefly dropped.

If the cause was a downstream dependency, confirm that dependency's latency has returned to baseline. A consumer whose offset advances only because the downstream call is now timing out fast has swapped a stall for a different failure.

Where it breaks

Backward compatibility is not forward compatibility. The registry approving a change is not evidence the change is safe for every consumer. It is evidence a new reader can read old records. The consumer pinned to the old reader schema is the one that breaks, silently, and the registry has no reason to flag it.

The lag metric stops at the topic. A stall caused by a slow HTTP call two hops downstream and a stall caused by a bad message produce identical lag charts. The metric cannot distinguish them, which is why step 1 and step 5 both exist.

Slow records get consumers evicted. If processing one record exceeds max.poll.interval.ms, the broker removes the consumer from the group and reassigns its partitions. The bad record then stalls a different consumer, then another, and the group appears to be crash-looping when one record is responsible.

The evidence moves. By the time someone investigates, a rebalance may have moved the stuck partition to a different consumer instance, so logs on the instance that first stalled may be the wrong logs.

Skipping the offset is a decision, not a fix. Resetting past a poison pill restores flow and drops a record. Whether that record mattered is a business question, and it should be recorded as skipped rather than lost.

The model only sees what is connected. NOFire AI can only walk the hop from consumer to dependency if that dependency edge is observed. A consumer whose downstream call is not instrumented shows as stalled with no visible cause, and the honest output there is that the edge is missing rather than an invented explanation.