Alert config debt they didn't know they had. Surfaced automatically across 1,190 investigations.
Ergeon installs fences, artificial grass, decks, and concrete for homeowners across 30+ US states. 13,700+ reviews, same-day video quoting. Behind the consumer product: 5,584 active infrastructure entities across EC2, EKS, Lambda, RDS, ElastiCache, and SQS. 1,817 EC2 instances. 983 Lambda functions across 20+ third-party integrations. 473 SQS queues. 12 RDS databases. When the backend has a problem, customer notifications stop, contractor scheduling stalls, and project timelines slip.
The challenge
Every alert started with the same question: is this real? Thresholds set during initial deployment hadn't been updated in over a year. The system had grown. The alerts hadn't. Real incidents and false positives were indistinguishable from the surface.
Queue-length monitors firing on thresholds that hadn't been updated since the original deployment. The system outgrew the alert config.
Failure-rate alerts claiming 100% breakdown while the system processed normally. The metric was miscalculated. An engineer still had to verify that manually every time.
A silent pod termination causing a queue backlog looked exactly like a misconfigured alert. Without correlating CloudWatch events with queue metrics, no way to tell.
Every alert: pull metrics, check logs, review deployments, decide whether to escalate. 30-40 minutes each, real or not.
Monitoring alerts pointed at services that no longer existed: deleted CDC entities, rotated DaemonSet pod names, thresholds misconfigured for non-existent infrastructure. Alert config debt had accumulated invisibly. No team has bandwidth to audit it.
How they use NOFire
Ergeon connected AWS, Prometheus, CloudWatch, and their Kubernetes cluster to NOFire. It runs alongside their existing Slack workflow. When an alert fires, NOFire checks what changed recently, tests hypotheses against live metrics and logs, classifies the alert, and posts a triage summary. Engineers review the answer instead of doing the investigation.
On every alert:
Recent changes checked first: deployments, pod events, scaling activity. Then hypotheses tested against metrics and logs. Worker crashes, slowdowns, task spikes, config drift.
False positives classified with confidence scores and evidence. Most engineers don't bother verifying.
Real incidents arrive with a diagnosis and the affected services. Context is already there.
Over 11 months:
1,190 investigations. The vast majority completed with a structured triage outcome.
False positives and misconfigured alerts closed without paging anyone. Alert config debt surfaced as a byproduct of daily triage.
Volume grew as engineers trusted the first-pass answer and stopped second-guessing. Routine alerts became a non-event.
The impact
700+ engineer hours back: 1,190 investigations at 30-40 minutes each if done manually. That time went to feature work and infrastructure improvements. Every month, roughly 60 more investigations run without anyone being pulled in.
False positives closed. Alert config debt surfaced.: Beyond catching individual false positives, NOFire revealed a pattern: alerts pointing at deleted services, stale pod names after DaemonSet rotations, thresholds misconfigured for non-existent entities. The audit no team has bandwidth for, done automatically as a byproduct of daily triage.
Same false positive, 14 times. Handled faster each time.: One queue-length alert fired 14 times on a stale threshold. By the third occurrence, the investigation was done in minutes. Engineers stopped treating it as real.
Scale incidents caught before cascade: A CDC event stream generated 4,964 concurrent Lambda invocations. A Django service spiked database query load 102x above normal. These are the incidents that take 90 minutes to untangle manually. Diagnosed before anyone finished reading the alert.
“We used to jump between 5 dashboards trying to piece together what happened. Now we get the full picture in one view. Our on-call engineers fix things instead of escalating to everyone.”Odysseas Tsatalos · CTO, Ergeon