Building Secure, Agent-Driven SRE Workflows with MCP

Before you start

Model Context Protocol (MCP) is the open standard, published at modelcontextprotocol.io, that lets an agent such as Claude Code or Cursor call external tools and read external data through one interface. For SRE work, the question is not whether to use MCP but which servers to connect, in what order, and with what permissions.

Map the servers you are considering against the job each one does. The landscape below is organised by what the server exposes, because that decides both its usefulness and its risk.

Server type	Examples	What the agent gets	Default risk
Incident and on-call	PagerDuty and Rootly MCP servers	Incidents, schedules, timelines, responders	Low if read-only; acknowledging and resolving are writes
Observability	Grafana, Datadog and other vendor servers	Metrics, logs, traces, dashboards	Low, but query volume and cost can surprise
Cloud and infrastructure as code	AWS, Terraform and Kubernetes servers	Resources, plans, cluster state	High when the token can apply, delete or exec
Source control	GitHub's MCP server	Repositories, pull requests, commits	Medium: merges and branch writes are production-adjacent
Production context	NOFire AI MCP server	Dependencies, recent changes, deployment risk, blast radius, incident history	Low: the token is read-only by design
Local-first open-source servers	Community servers on GitHub	Varies: often logs, kubectl or shell wrappers	Varies: read the tool list before connecting

Check two things for every server before connecting it: whether its token can write, and whether its tool definitions come from a source you trust. The OWASP MCP Security Cheat Sheet treats tool definitions as untrusted input for good reason.

Recommended MCP servers for incident response and where to get them

These five servers cover the context, incident, cluster, code and observability jobs in the table. Each one can run without write access.

• NOFire AI MCP server. NOFire AI is a production ops platform for engineers and AI agents. Its MCP server gives the agent the production map (a live, sourced model of services, dependencies and changes) through nine read-only tools. nofire_analyze_blast_radius returns the affected services by hop depth. The server is hosted at https://mcp.nofire.ai/mcp, so there is nothing to download. Connect the NOFire MCP server has the setup.
• PagerDuty MCP server. PagerDuty's official server exposes incidents, services, schedules and event orchestrations. PagerDuty hosts it, with OAuth or API-key access, and the setup steps are on PagerDuty's MCP server page. To limit what it can do, use a scoped OAuth client. The open-source local version, PagerDuty/pagerduty-mcp-server, is now archived.
• Kubernetes MCP server. containers/kubernetes-mcp-server on GitHub (Apache-2.0) exposes Kubernetes and OpenShift clusters. It installs through npm or PyPI. Its setup guide runs it under a dedicated read-only ServiceAccount with a short-lived token.
• GitHub MCP server. github/github-mcp-server is GitHub's official server (MIT license) for repositories, issues and pull requests. GitHub hosts a remote version, or you run it locally. With the --read-only flag, it offers only read tools.
• Grafana MCP server. grafana/mcp-grafana on GitHub (Apache-2.0) exposes dashboards, datasource queries such as Prometheus and Loki, and Grafana Incident. The --disable-write flag runs it read-only.

If you run a server locally, install it only from the publisher's own repository or documentation, and pin the version you reviewed.

The steps

Establishing production context

An agent has production context when it can answer what a service depends on, what changed on it recently, and what fails if it goes down. MCP servers that expose raw signals do not provide this on their own, because the relationships between signals are not in the signals. Connect a context source first, then the signal sources it helps interpret.

1. Connect a production map. A production map joins Kubernetes topology, cloud configuration, infrastructure-as-code state from Helm and Terraform, deploy history and observability signals into one model with dated edges. Connect the NOFire MCP server covers the token and client configuration. The agent gets nine read-only tools, including dependencies, recent changes and blast radius.

2. Add signal servers read-only. Connect your observability and incident servers with read-only tokens. With context already connected, the agent can ask which of fifty alerting services sits upstream of the others instead of summarising all fifty.

3. Prefer local-first servers for sensitive data. Where a community server wraps kubectl or log access, run it locally under the developer's own scoped credentials rather than exposing a shared cluster token.

Implementing automated incident analysis

In automated incident analysis over MCP, the agent pulls three things. It takes the alert from the incident platform, the dependency and change history from the production map, and the supporting metrics from observability. It then names a candidate cause. Deterministic context is what turns three tool calls into an investigation. Without it the agent returns a pile of output.

4. Give the agent a fixed opening sequence. For any page: fetch the incident, search the affected entity, list its changes in the last few hours, then analyze its blast radius. NOFire AI's server exposes each step as a tool, so the agent follows the dependency path rather than guessing.

5. Ask for blast radius before any fix. Before the agent proposes a rollback or restart, have it call the blast radius tool on the target. Across distributed systems, reach is the fact the codebase alone cannot supply.

6. Keep investigation and remediation separate. Use MCP for the investigation and hand the proposed fix to a person or a governed pipeline. NOFire AI's investigation links every claim to the deploy, config event or trace behind it, so the proposal is checkable.

Securing agent-driven remediation

Securing remediation means no agent action changes production without passing a control the agent cannot turn off. MCP defines how an agent calls a tool. It does not decide whether the call should be allowed. That decision needs a policy layer, an authorization boundary per token, and isolation around the agent process.

7. Scope every token. One token per person and tool, with the narrowest permissions the server supports. Rotate on a schedule.

8. Put a policy gate in front of write-capable servers. Route write tools through an enforcement point that evaluates each call against policy and predicted blast radius and can refuse it. What governing an agent that acts requires covers the verdicts and the signed record.

9. Sandbox the agent. A prompt-injected agent with a valid token does what the injection says. Running the agent session inside a microVM bounds what it can reach on the host. brig is NOFire AI's open-source sandbox for coding agents on macOS and Linux, and how to sandbox an agent with write access covers the substrate choice.

Agent rules, workflows and MCP security

Agent rules are instruction files an agent reads at the start of a session. Claude Code reads CLAUDE.md, Cursor reads rule files in .cursor/rules, and many other coding agents read AGENTS.md. For remediation, three layers work together. MCP servers supply the tools and the data. Agent rules set the order of calls and the limits. A workflow runs fixed instructions on a trigger, such as an alert or a push, so no one has to start the run. NOFire Workflows run this way against the production map.

10. Write the investigation sequence and the limits into the agent rules. Put the sequence from step 4 in CLAUDE.md, AGENTS.md or a Cursor rule. Name the servers that are read-only, and state that every fix goes to a person or a governed pipeline. A rule is text in the agent's context, so a prompt injection can override it. Enforce the same limits with token scope and the policy gate from steps 7 and 8.

11. Check whose permissions a server uses. A server that calls tools with its own broad credentials gives every connected agent those rights. The OWASP cheat sheet calls this the confused deputy problem. Prefer servers that authorize as the calling user through OAuth or a personal token, so the agent can do no more than its user.

12. Treat tool descriptions and tool output as untrusted input. Tool poisoning means hidden instructions in a tool's description, schema or output. Prompt injection reaches the agent the same way, through an alert body, a log line or a ticket comment. Give each server only the permissions its job needs, and keep write tools behind the policy gate. An injected instruction then meets a check before it reaches production.

13. Keep governance controls outside the agent. Log every tool call with the caller, the arguments and the result, in a store the agent cannot write to. Require a person to approve any write that policy does not allow on its own, and keep a way to revoke an agent's tokens at once. AI agent audit and compliance covers what the record must hold.

Verify it worked

Ask the agent to list its tools and confirm each connected server appears with the tools you expected, and no write tools you did not intend.

Run a real investigation on a resolved incident. The agent should fetch the incident, name the entity, list its recent changes and return a blast radius without being told which tool to use.

Attempt a write through a read-only token. It should fail at the server, not at the agent's discretion.

Where it breaks

Malformed client config drops every server. A trailing comma in the MCP config file silently loads nothing, which looks like every integration failing at once.

Signal without context produces summaries, not causes. An agent connected only to observability and incident servers describes the alert storm well and rarely names the change behind it.

Tool definitions are an attack surface. A community server whose tool descriptions change after install can steer the agent. Pin versions and review the tool list on update, as the OWASP MCP guidance recommends.

Read-only is a property of the token, not the protocol. An MCP server is exactly as safe as the credential behind it. Check each one.