Alert triage

At a glance

Stage	Question it answers	Automatable today
Deduplication	Have we already got this one?	Fully, on a stable alert identity
Grouping	Do these fifty alerts describe one failure?	Largely, using time and topology
Enrichment	What changed near this service recently?	Fully, given deploy and config history
Impact scoring	Who or what is affected, and how much?	Given a dependency graph, yes
Cause hypothesis	What started it?	Depends on the model, see the accuracy question
Routing	Who should hold this?	Fully, if ownership is inferred rather than declared

How it works

Triage is a sequence of narrowing decisions, and most of them are mechanical. Deduplication collapses repeats of the same condition, which requires a stable identity for an alert rather than a message string that changes with every firing. Grouping collapses alerts that describe one failure, which needs both time proximity and a topology to say the affected services are actually related. Without the topology, grouping by time alone merges unrelated incidents that happen to overlap.

Enrichment attaches what changed near the affected service in the relevant window: deploys, config updates, scaling events, schema migrations. This is the step that most often turns a page into a five-minute resolution, and it needs no inference at all, only a change history joined to a service.

Impact scoring is where triage stops being about the alert and starts being about the system. An error rate on a service carrying three percent of direct traffic sounds minor until the dependency graph shows it is a synchronous dependency of something carrying sixty. That calculation is the same one described in what blast radius is, applied to prioritisation rather than to bounding an action.

Routing closes the loop. Ownership inferred from deploy history, contributor activity and on-call rotation stays current, where ownership declared in a file drifts the first time a team reorganises and nobody edits it.

How it is measured

Three numbers describe a triage system honestly, and the first two are usually reported without the third.

Page volume is the easy one and the most misleading. It falls whenever anything is suppressed, including things that mattered. Actionable rate, the share of pages that led to a change, is better, because it measures the quality of the decision rather than its aggressiveness. The one that keeps a triage layer honest is the miss rate: incidents that the system saw signal for and did not escalate. It is harder to collect, because it can only be counted retrospectively from incidents that were found another way, and it is the number that tells you whether the volume reduction was earned.

Time to first meaningful context is worth tracking alongside those. Between the page firing and the responder knowing what changed recently near the affected service, most teams spend several minutes on lookups a triage layer can do in advance.

Common misconceptions

That fewer pages is the goal. Fewer pages is trivially achievable and says nothing on its own. The goal is that the pages that fire are the ones a person needed to see.

That grouping tells you the cause. A group of fifty alerts is one incident described fifty times. Which member of that group is the origin is a separate question, answered in automated root cause analysis.

That a quiet week means it is working. A quiet week is also what a broken alerting pipeline looks like. Triage systems need a liveness check for the same reason the monitoring underneath them does.

That the rules can be left alone once tuned. Every suppression rule is a bet about a system that keeps changing. A rule written to silence a noisy service stays in place after that service is rewritten, and the failure it then hides is one nobody remembers deciding to ignore. Rules that cannot expire accumulate into a triage layer whose behaviour nobody can describe, which is the state most teams are actually in when they go looking for automation.