We test every change to the NOFire AI investigator on SREGym, an independent Kubernetes benchmark, paired by problem.
One agent against a team of agents, 105 problems, 3 attempts each, same model, no Kubernetes tools
Same model, same tools, same Production Context Graph. We replaced a planner, workers, an evaluator and a writer with one agent loop and a ledger of evidence. Both designs ran on the same 105 problems, 3 attempts each.
The model on both sides was Claude Sonnet 4.6. Ranges are 95% intervals that resample whole problems.
Pass rate of the new version over the old, paired by problem, with 95% intervals. A dot right of 1.0x is a gain. A line that crosses 1.0x is not a measured change.
| Change | Problems | Pass rate, new over old | 95% interval | Clear of 1.0x |
|---|---|---|---|---|
| One agent instead of a team, no Kubernetes tools | 105 problems | 1.5x | 1.3x to 1.7x | Yes |
| One agent instead of a team, with Kubernetes tools | 105 problems | 1.3x | 1.2x to 1.4x | Yes |
| One agent instead of a team, problems never used in tuning, no Kubernetes tools | 84 problems | 1.5x | 1.3x to 1.8x | Yes |
| One agent instead of a team, problems never used in tuning, with Kubernetes tools | 84 problems | 1.25x | 1.1x to 1.4x | Yes |
| Newer model, same system | 21 problems | 1.10x | 0.92x to 1.32x | No |
| Haiku 5.5 instead of Sonnet 5.5, same system, with Kubernetes tools | 102 problems | 0.93x | 0.87x to 0.99x | Yes |
| More reasoning effort, 3 settings | 21 problems | On the score, no setting was significant after the Holm correction. | ||
Every handoff between agents was a summary, and each summary dropped detail. One agent holds every tool result in one context, gives each result an id and ends with a report that code checks. Every claim in the report must cite evidence the run recorded.
Read the post
SREGym gives a grade for the whole investigation. These are the rules we follow when we test a change.
SREGym comes from Tianyin Xu's lab at UIUC. SREGym deploys an application on Kubernetes, injects a fault and asks the agent for the root cause. The SREGym judge scores each answer from 0 to 1, and an answer passes at 0.70. In the runs that compared one agent with a team, the judge was Claude Sonnet 5, never the model under test.
We tune on 21 problems and confirm on 105, and 84 of the 105 were never used in tuning. Each problem gets 3 attempts, every attempt is scored, and there is no best of 3. Attempts on one problem tend to agree, so 63 attempts are worth about 33 independent ones. Intervals resample whole problems.
Before comparing two versions, we compare one build with the same build. Across eight pairs of runs of the same build, the pass rate moved by up to 20 points with nothing changed. Cohen's kappa between the two runs ran from 0.34 to 0.73. In the worst pair, an image pull failure in the second run looked exactly like a regression.
Nothing changed, and the pass rate moved by up to 20 points
The test, the metric and the decision rule are fixed before the run. With 21 problems and 3 attempts, the smallest change a sign test detects 80% of the time is about 32 points. So a run of 21 problems can screen an idea but cannot accept one. Spending attempts on more problems beats more repeats.
Spend attempts on problems, not repeats
Cost is the model spend at list prices. Time runs from the problem to the report. Failed investigations count in both. A judge step once raised the cost by 93% with no change in accuracy, and only the cost column showed the bug.
In production posture the agent reads metrics, logs, traces and the change history NOFire AI records, with no direct access to the Kubernetes API. Our customers run this way. In the second posture the agent also gets read-only Kubernetes tools.
A judge from a different model family scored 376 attempts. Agreement with the SREGym judge was a kappa of 0.66. A result that changes with the judge is not yet a finding.
SREGym is open source. When our runs hit a problem in the harness, we fixed the problem upstream. These fixes are merged:
