Context · Kafka

A broken message. A stalled consumer. Both named, not guessed.

NOFire tells a broken message from a slow consumer, and a stopped consumer from a scaling problem, then ties either to the service, team, and change behind it. Read-only, from outside your apps, no code changes.

Broken messages caught before they cascadeStopped vs. slow, told apart automaticallyBlast radius named, not guessed
app.nofire.ai/investigationsLive

Schema compatibility mode changed from BACKWARD to NONE

INV-1
What happenedTheoriesEvidence 13ThreadsActions 1Postmortem

Consumer group settlement-group froze with offsets stuck at 516/257 and lag growing at 80 msg/min. Root cause: schema compatibility changed from BACKWARD to NONE, allowing an incompatible v2 schema the consumer cannot deserialize. Silent stall: no errors, no alerts, logs simply stopped.

orders-events-value compatibility mode

Event sequence

  1. symptomJul 10 10:45 PM

    Schema Registry connection failures begin, 12 occurrences over 34s

  2. symptomJul 10 10:56 PM

    Schema Registry recovered, serving HTTP 200 responses

  3. symptomJul 10 10:58 PM

    settlement-consumer logs stopped, last v1 event consumed at offset 515

  4. changeJul 10 10:59 PM

    Schema compatibility changed BACKWARD→NONE at version 2 for orders-events-value ← ROOT CAUSE

  5. impactJul 10 11:02 PM

    Consumer offsets frozen: partition 0 at 257, partition 2 at 516, lag growing

Impact: root (red) affected (amber)

settlement-consumersettlement-group

1 service, 1 consumer group, 3 partitions frozen

The challenge

A black box the business runs on.

Transactions, user events, inventory, financial decisions: most critical business outcomes travel through topics and pipelines. When that layer is ungoverned, the blast radius is not an engineering incident. It is wrong data powering business decisions, and compliance exposure.

Deep dive: poison pills and lag, root-caused

Silent failures

Every broker dashboard is green while a consumer sits stalled and orders quietly pile up. The number you alert on does not tell you a consumer stopped.

One bad message, no warning

A malformed payload, an incompatible schema, a producer bug: Kafka engineers call this a poison pill, and it freezes a consumer with no error and no owner in sight, whichever team caused it.

Lineage nobody can produce

Who consumes orders? Where does PII flow when a regulator asks? Can we retire this topic? A group.id names none of it, so the answer lives in someone's head.

How it works · Flow health

Know the health of the flow, not just the brokers.

Real-time revenue rides Kafka. When a consumer stalls, the brokers stay green and the backlog grows in silence. NOFire watches the flow end to end and reads what the lag actually means.

app.nofire.ai/investigationsLive

Schema compatibility mode changed from BACKWARD to NONE

INV-1
What happenedTheoriesEvidence 13ThreadsActions 1Postmortem
10:04 PMCHANGES

Lag growing continuously after freeze: partition 2 lag increased from 265 to 585 messages (80 msg/min growth rate), partition 0 lag increased from 133 to 293 messages (40 msg/min growth rate)

10:04 PMSTATE

Consumer group settlement-group consuming from topic orders-events with 3 partitions, configuration validated as correct

10:57 PMLOGS

Log silence detected: service stopped logging at 21:58:54Z, 5 minutes before investigation window end, indicating silent stall during incident

{pod=~".*settlement-consumer.*"}
10:58 PM consumed v1 order id=884772 at offset 515
10:58 PM consumed v1 order id=884771 at offset 514
10:58 PM consumed v1 order id=884770 at offset 256

View Full Logs

  1. 01

    Map the flow, no code changes

    NOFire reads the cluster from the broker itself and follows each consumer to the service behind it. No agents in your apps, no libraries to ship.

  2. 02

    Read lag as evidence

    Committed offset against the topic head, sampled over time, tells a stopped consumer from a slow one: partition 2 growing at 80 msg/min under a frozen offset is a stall, not a scaling problem.

  3. 03

    Catch the silent stall

    No errors, no alerts, logs simply stop. NOFire detects the log silence, ties it to the frozen offsets, and names what the stall puts at risk before customers feel it.

How it works · Poison pill

One bad message breaks the consumer, whoever sent it.

A malformed payload, an incompatible schema, a bad producer deploy: any of them can freeze a downstream consumer with no error. NOFire names the message and the change behind it, whichever team it came from.

app.nofire.ai/investigationsLive

Schema compatibility mode changed from BACKWARD to NONE

INV-1
What happenedTheoriesEvidence 13ThreadsActions 1Postmortem

Postmortem · generated from investigation findings

Regenerate

settlement-consumer frozen due to schema incompatibility

The root cause was a schema compatibility mode change from BACKWARD to NONE, which allowed registration of an incompatible v2 schema the consumer could not deserialize. The consumer stalled silently, with no error handling or instrumentation to surface the issue.

Learnings

Coordinate schema compatibility mode changes

Edit

Before changing compatibility mode from BACKWARD to NONE, verify all consumers can handle the new schema or coordinate a deployment. Require consumer team sign-off for compatibility mode downgrades.

Add deserialization error handling and alerting

Edit

Surface deserialization failures as alerts to the on-call team immediately, rather than silently stalling offset commits.

Monitor offset commit lag and stalls

Edit

Alert on offset commit stalls and rapid lag growth. These are early indicators of deserialization or processing failures that may not surface in application logs.

  1. 04

    Every bad message tied to its source

    A new schema version, a malformed payload, or a bad deploy is linked to the exact consumer it broke, across team boundaries, so the blast radius is known, not guessed.

  2. 05

    This poison pill, connected

    Here, a compatibility mode dropped to NONE and an incompatible schema landed, freezing the consumer with no error. The same mechanism catches a malformed payload or a bad deploy just as fast.

  3. 06

    Written back so it doesn't repeat

    The postmortem generates the guardrail the failure needed: consumer sign-off for compatibility downgrades, alerts on commit stalls, deserialization failures surfaced automatically. Learned once, enforced after, whoever causes the next one.

How it works · Estate map

Every group, topic and consumer, on one live map.

Hundreds of topics, opaque group.ids, and the answer to who consumes what living in one busy engineer's head. NOFire turns the estate into a map that stays current on its own.

app.nofire.ai/exploreLive
events-prodKafkaCluster
Now
Application MapEdgesHealthyDegradedErrorsInferred
CONTAINS-TOPICCONTAINS-TOPICCONTAINS-BROKERCONTAINS-TOPICCONTAINS-TOPICCONTAINS-TOPICevents-prodKAFKACLUSTERpayments-eventsKAFKATOPICinventory-eventsKAFKATOPICbroker-0KAFKABROKERorders-eventsKAFKATOPICorders-dlqKAFKATOPICaudit-eventsKAFKATOPIC
  1. 07

    First-class entities, one graph

    Clusters, brokers, topics, and consumer groups sit in the same production context graph as your services and deployments, with the relationships drawn: contains-topic, consumes-from, deployed-on.

  2. 08

    A map that stays current

    The graph updates as your estate changes, and it is time-versioned: ownership, schema versions, consumer health, and lineage, answerable at any moment, not just when the pager fires.

  3. 09

    One model, every Kafka

    MSK, Confluent, Strimzi or self-hosted, the same read-only model spans them all, so a mixed estate reads as one.

Why NOFire

Exporters hand you a number. NOFire hands you the cause.

kafka-exporter, Burrow, CloudWatch
  • A lag number with no owner
  • No idea if the consumer stopped or slowed
  • No link to the change that caused it
NOFire
  • The named service, team and deploy behind the group
  • Stopped or slow, read across samples
  • The change that caused it, with evidence and blast radius
Runs on the Production Context Graph

Every consumer, topic, and schema change lands in the same live model your investigations run on: services, dependencies, ownership, and change history in one graph.

See the Service Catalog
What you get

Change production, keep it running.

Provider agnostic

One model for MSK, Confluent, Strimzi and self-hosted Kafka.

Read-only by design

No write access to production. NOFire reads cluster metadata, never touches your data.

No payload access

Diagnosis uses metadata calls only. NOFire never consumes your records, so it never sees their contents.

AI reasons, humans decide

NOFire reasons over the evidence and puts the cause forward. Your engineers decide what to do about it.

Faster root cause

Hours of cross-tool digging become minutes, with the evidence attached.

Lineage, answerable

Ownership, schema versions, consumer health, and lineage in one place, when an incident or a regulator asks.

AI writes the code. NOFire keeps it running.

A 30-minute call with a founder. We map your stack to the Context & Control Model, live.

Book a demo