Why this decision now.
Your team ships more than it did two years ago. The part that did not speed up is everything after merge: the alert at 02:00, the hour spent deciding whether a deploy caused it, the second hour spent proving it to someone else.
The expensive part of an incident is rarely the fix. It is the assembly of context that has to happen before anyone can think. When context is hard to assemble, teams escalate instead, and one engineer's memory quietly becomes a piece of production infrastructure.
Two things changed at once. Production now changes faster than anyone can model by hand. And agents already act in it: they open pull requests, run test suites, and hold credentials in CI. That second change matters for a specific reason. An agent produces its own log of what it did, written by the thing being investigated. It is a useful signal and a weak witness.
Strip the category language out and a working system does three things. It names the change. It shows the evidence, in a form you can open in your own tools. And it leaves a record of anything it did, produced independently of the agent that did it.
Build vs buy (and the other paths).
A capable engineer wires a model to your logs on a Saturday. By Sunday it is summarising incidents and the summaries are genuinely good. The instinct after that weekend is that the remaining work is small and mostly polish. That instinct is the most expensive assumption in this category, because the prototype worked on a narrow problem, chosen by the person who understood it best, against incidents that were already resolved. Every one of those conditions changes on the way to production.
Five cost layers follow the prototype, and the two that end projects are the last two: model access priced by your alert noise rather than your results, infrastructure, people, evaluation debt, and model churn. A sixth applies only once agents act rather than read, which is what constrains a wrong action, and it is decided before the first write rather than hardened afterward.
Underneath the layers sit three commitments rather than one project. Build has an end date and every proposal costs it correctly. Operate never closes. Improve is the one most teams never start, and without a way to score changes the system plateaus below the bar at which people trust it.
Five paths, not two.
Do nothing, build in-house, wrap a model, extend your observability vendor, or buy a production tool. Build is a real answer for teams who are the platform, or who have constraints no vendor will honour. It is the wrong answer if the reason is that the weekend demo looked close.
This chapter is deliberately short, because the decision has its own guide. Build vs Buy for AI in Production carries the full treatment: all five paths stated at their strongest, the three expertise gaps, and the opportunity cost.
How to evaluate.
Two questions to ask before you see a slide. They are hard to rehearse and they tell you more than the demo will.
When the first answer is wrong, what happens next?Gate 1
Every system will be wrong. The interesting property is what the wrong answer costs you. Does it show its reasoning so you can correct the step that went wrong, or does it restate the same conclusion with more confidence? Can it say it does not know?
When an agent runs a command, who has the record, and was it produced outside the agent?Gate 2
If the only account of what an agent did comes from the agent, you have logging, not evidence. Ask where the record is written, what produces it, and whether the agent can alter it.
Six properties, each with a test.
Knowledge of how production is wired. Reasoning you can follow and correct. Safe action under approval. Learning from past incidents. Works where the team works. And control: boundary, record, approval, reversibility. Each one in the guide carries a definition, a test you run, and the failure mode to watch for.
Checks that break a rehearsed demo.
Ask for an incident where one telemetry source was wrong. Rename a field the system depends on. Ask for an answer in ninety seconds. Give it two incidents with similar symptoms and different causes. Tell it the cause is something it is not, and see whether it argues with evidence or agrees with you.
The trial.
A hosted environment first, so you can see the shape of it without integration work. Then four to six weeks on your own systems, because the only meaningful test is your stack, with your noise, at your scale. Write the exit criteria and the decision date before it starts. A trial with no written exit criteria becomes a pilot with no end.
Making the decision.
The same purchase looks different to each stakeholder. Answer all four or the decision stalls late.
- Platform engineering asks what this adds to the thing they already carry.
- SRE and on-call ask whether it helps at 02:00. If it adds a console to check during an incident, it is a cost, not a tool.
- Security and compliance ask what happens when an agent holds a credential.
- Finance asks what it replaces, in numbers from your own environment.
Score things you can measure before and after in your own systems: time from alert to a named and evidenced cause, share of incidents resolved without escalating, share of investigations where the first answer held up, agent actions with a complete out-of-band record, and how many people can run an investigation unaided.
Deliberately absent from that list: percentage reductions in mean time to resolve. It is easy to move by redefining the measurement window, and it will not survive scrutiny from the room that approves the spend.
The evaluation worksheet.
The appendix is built to be printed and taken into a vendor call. Twenty rows across four blocks: company and fit, investigation, agents and control, integrations and trial. Mark each Yes, Partial, or No, and write the evidence rather than the impression.
Eight rows are must-pass, and three of them sit in the control block:
- A named reference at your scale and regulatory exposure.
- Evidence you can open in your own tools.
- Correlation demonstrated across two tools, not within one.
- A record of what agents actually did, produced outside the agent.
- Saying "we don't know yet" is a supported outcome.
- Where actions are allowed to run: isolation per task, or an explicit read-only scope in writing.
- You know what data leaves your environment, and when.
- A trial on your systems with written exit criteria and a decision date.
A No on any of the eight is not a negotiating position. For a system that will act in production, it is a stop. The control block is weighted deliberately: the other blocks describe capability that can improve after you buy, while that one describes where the system sits relative to your trust boundary, which is decided once.
The worksheet is vendor-neutral on purpose. Hold every vendor to the same eight must-pass rows, including NOFire. For the isolation model behind our own answer to the control rows, see Sandboxing for Agentic Execution. For the path comparison in Chapter 2, see Build vs Buy.