What is AI agent governance?

AI agent governance is the set of controls, policies, and audit mechanisms that constrain what AI agents running unattended are allowed to do, especially in production systems where an agent can deploy code, change configuration, or remediate incidents. Effective governance enforces policy at runtime, not only after the fact. The distinction matters because an agent acting on live infrastructure can cause irreversible damage in seconds.

NOFire AI is a runtime security policy enforcement platform for AI agents acting on production infrastructure. It evaluates every action against policy before execution, attaches the predicted blast radius to the verdict, and signs both as tamper-evident audit evidence.

Why audit is not enforcement

Most platforms let an agent act and log it afterward. That is audit, not governance.

Logging tells you what went wrong. It does not stop it. By the time a log entry records that an agent restarted a fleet of database replicas or pushed a schema migration to production, the damage is done. Audit trails are necessary for forensics and compliance, but they carry no enforcement authority.

Governance means the policy refuses an out-of-bounds action at execution time, with the predicted blast radius attached. The agent receives a rejection, not a permission, before the call is made. That verdict and the predicted impact become the audit record, so the evidence is a byproduct of enforcement rather than a separate system bolted on afterward.

Incident remediation agents, deployment pipelines, and configuration management tools increasingly act on live infrastructure with minimal human review. A system that only logs what they did is not governing them.

Which frameworks expect AI agent governance

No framework names a product, and none of them uses the phrase "runtime enforcement". All four ask you to demonstrate control over an autonomous system and to produce evidence that the control was applied.

NIST AI RMF (NIST AI 100-1). The voluntary US framework, published January 2023, organised into four functions: GOVERN, MAP, MEASURE and MANAGE. GOVERN is the cross-cutting one covering policy and accountability, and MANAGE covers responding to and remediating identified risks. The Generative AI Profile (NIST AI 600-1, July 2024) extends it to risks specific to generative systems. A runtime policy layer is how the GOVERN function becomes a mechanism rather than a document: the policy is the stated control, and the signed verdict is the evidence it ran.

ISO/IEC 42001:2023. The first international standard for an AI management system, and the one an organisation can actually be certified against. It asks for established, implemented and continually improved controls over how AI is developed and used. Runtime enforcement records supply the operating evidence a certification audit looks for.

The EU AI Act. For high-risk systems, Article 12 places a design-time obligation on the provider to build in automatic event logging with traceability across the system's lifetime. Article 26(6) places the matching runtime obligation on the deployer, who must keep those logs for at least six months unless other Union or national law requires longer.

SOC 2, criterion CC7. System operations: detecting anomalies, responding to incidents, and showing an auditor that monitoring was active throughout the observation period rather than merely configured.

NOFire AI is not a certification and does not grant one. What the enforcement layer produces is the evidence these frameworks ask for: a signed, ordered record showing which policy applied, what it decided, and what the action would have affected. What an auditor actually does with that record, and the four properties that decide whether it holds up, is a separate question from which framework asked for it.

What to log to prove an agent's actions

For each agent action, record:

1. The policy verdict. Permitted, refused, or permitted with modification, and the identifier of the specific policy that decided.
2. The tool call. The tool or API invoked and the arguments passed to it.
3. The predicted impact. The blast radius computed before execution, so the record shows what was known at decision time rather than what happened afterwards.
4. The bound identity. Which agent instance, role and session acted. An action attributed to "the agent" is not attributable.
5. Human approval, where required. Who approved, when, and against which verdict. Absence of an approval is itself a fact worth recording.
6. A timestamp and a signed hash. Each record chained to the previous one, so a gap or an edit is detectable rather than invisible.

Article 12 of the EU AI Act is the requirement to be capable of this, and it sits on the provider. Article 26(6) is the requirement to retain it, sits on the deployer, and sets six months as the floor rather than the target. SOC 2 CC7 asks a different question of the same records: not that they exist, but that they show continuous operation across the whole observation window.

The practical failure is not missing logs. It is logs that record the action and not the decision, which prove an agent did something and cannot prove anything was governing it.

The OWASP agentic threat classes

OWASP is where the agentic threat taxonomy actually lives, across the Agentic AI Threats and Mitigations work, the OWASP Top 10 for Agentic Applications, and the LLM Top 10. Runtime enforcement addresses some of these classes and not others. The four below say which.

Tool misuse. An attacker manipulates an agent into invoking a legitimate tool in an unintended way. Enforcement helps directly: scope which tools a given identity may call, and cap the predicted impact of each call.

Excessive agency. OWASP splits this into excessive functionality, excessive permissions and excessive autonomy. All three are policy questions rather than model questions, which is why a better model does not fix them. Least-privilege tool grants address the first two. A human approval requirement on high-impact actions addresses the third.

Privilege compromise. Weak permission management exploited through role inheritance or misconfiguration. Identity binding and least-privilege grants scoped to a single task are the controls, and they only work if the grant is time-limited.

Supply chain. Compromise arriving through a dependency, model, or tool definition. This is the class runtime enforcement is weakest against, and saying so matters. Sandboxing constrains what compromised code can reach once it is running. It does nothing about verifying the artifact before it loads. That needs signature verification in the pipeline, upstream of anything on this page.

From policy verdict to OpenTelemetry trace

Governance and observability are different jobs, and the useful connection between them is a data one.

An agent action already produces a trace: the span for the tool call, its arguments, its latency, its outcome. A policy verdict is another attribute on that same span. Recording the deciding policy, the decision, the predicted blast radius and the bound identity as span attributes means the governance record travels with the execution record instead of living in a separate system nobody correlates under pressure.

That gives you one thing the two halves cannot give separately: for any incident, the ability to ask which agent actions were permitted in the window, what each was predicted to affect, and which were refused. A refusal is invisible to conventional observability, because nothing executed, and it is often the most informative event in the trace.

OpenTelemetry's semantic conventions for generative AI are still moving, so pinning to specific attribute names is premature. The stable part is the shape: the verdict belongs on the span, not in a parallel log.

Least privilege for agents

The IAM principle transfers, with one difference that matters.

For a human, least privilege is mostly durable: a role is granted, reviewed periodically, and revoked on departure. For an agent, the useful grant is per task and time-limited, because the agent's next task may be unrelated and its blast radius unbounded by anything it did before. A standing role broad enough to cover every task an agent might attempt is a permanent grant with a review meeting attached, whatever it is called.

In practice that means three things. The grant is scoped to the tools a specific task needs. It expires. And it is bound to a verified identity, so an action is attributable to an agent instance and session rather than to a shared service account, which is the single most common way agent actions become unattributable after the fact.

Production readiness checklist

Before running agents unattended against production:

1. Every agent action is bound to a verified identity, not a shared service account.
2. Tool grants are scoped per task and expire, rather than standing.
3. Every action carries a predicted blast radius computed before execution.
4. A defined impact ceiling exists, and actions above it are refused rather than logged.
5. Irreversible operations are classified as such and require explicit approval.
6. High-impact actions require a human signature, with the approver recorded.
7. Policy verdicts, including refusals, are signed and chained.
8. Audit records are retained for at least the six months the EU AI Act sets as a floor.
9. A kill switch halts all agent activity without requiring agent cooperation.
10. Policy verdicts are correlated with execution traces, not stored separately.

An item you cannot demonstrate on demand is not a control, whatever the policy document says. How to implement an AI agent governance framework turns this checklist into steps, with the verification tests for each.

The runtime policy primitives

Production-grade governance is not a single check. It is a concrete set of enforcement primitives that can be applied to any agent action:

Blast-radius bounds. Every action carries a predicted impact scope before execution. The policy layer defines a ceiling, for example, no more than 5% of traffic affected by a single action, and rejects actions that exceed it.

Schema-drift refusal. Agents that interact with databases or configuration stores can detect and refuse changes that would produce schema drift relative to a known-good baseline. This prevents cascading failures from silent structural changes.

Network-boundary gates. Agents are restricted to defined network boundaries. An agent granted access to a staging environment cannot reach production endpoints unless the policy explicitly permits it.

Sandbox grants. Untrusted or new agent code executes inside an isolated sandbox before receiving production access. The grant is scoped, time-limited, and attached to a specific task. Which isolation substrate to use, from container through gVisor and Kata to microVM and unikernel, is a separate decision from the grant itself.

Signed audit. Every action, along with the policy verdict and predicted blast radius, is signed at the point of enforcement. Signed records cannot be retroactively altered and provide audit-grade evidence for compliance and incident review.

Reversible-by-default actions. The policy layer prefers reversible actions and flags irreversible ones for elevated review. Destructive operations, for example dropping a table or terminating a process, require explicit approval that the action cannot be undone.

Quorum gates. High-risk actions require agreement from multiple policy authorities before execution. A single agent decision is insufficient. A quorum verdict is required. Where the policy requires a human signature rather than another authority, the approval and the approver become part of the signed record.

Identity binding. Every agent action is bound to a verified identity, a specific agent instance, role, and session, so the policy layer knows who is acting, not just what is acting.

Kill switch. The governance layer includes a reliable mechanism to halt all agent activity immediately, without requiring the agent's cooperation.

See the Runtime Policy Patterns for the full set, with examples of how these primitives compose into a complete policy.

Governance vs agent-building

Most content labeled "AI agent governance" today covers how to build agents on Vertex AI, Azure AI Foundry, or similar platforms: prompting patterns, tool definitions, memory architectures. That is agent engineering, and it matters for building reliable agents.

Production governance is a different problem: controlling agents that already act on live infrastructure, often agents you did not build, running unattended on tasks that modify state you cannot easily roll back.

The governance layer sits between the agent and the environment. It intercepts action requests, evaluates them against policy, and either permits, modifies, or refuses them. The agent's internal design is largely irrelevant at this layer. What matters is the action it attempts to take and whether that action falls within the bounds the policy defines.

Who needs AI agent governance

Any team deploying AI agents that can modify production systems needs runtime governance. The most common cases:

Deploy pipelines. Agents that trigger rollouts, promote builds, or manage feature flags can affect millions of users in a single action. Blast-radius bounds and quorum gates are the minimum viable controls.

Incident remediation. Unattended SRE agents that restart services, reroute traffic, or roll back deployments are acting under time pressure with incomplete information. Policy enforcement prevents a remediation action from becoming a second incident.

Configuration management. Agents that modify application configuration, infrastructure-as-code, or secrets need schema-drift refusal and identity binding to prevent silent misconfigurations.

Database operations. Schema migrations, data backups, index changes: these are irreversible or expensive to reverse. Reversible-by-default policy and quorum gates provide a meaningful check before execution.

The common thread is irreversibility. Any agent that can take an action that is expensive or impossible to undo is a candidate for runtime governance, regardless of how well the agent was built.

See the Runtime Policy Patterns to go deeper on how these primitives are structured and applied, and what blast radius is and how it is bounded for the measurement the impact ceiling depends on.