Resources · 03
AI agent governance: a production readiness guide
Frame permissions, data, tests, oversight and shutdown before an agent is trusted with a business process.
· 4 min
What this guide helps achieve
- Limit the agent to what it needs
- Test costly failures before production
- Keep useful traces without over-collecting
- Maintain effective human accountability
Quick check
- Can the agent trigger an irreversible action?
- Is the submitted data authorised?
- Does a threshold require human approval?
- Do tools have minimum permissions?
- Can the system stop and roll back quickly?
Step-by-step method
- 01
Describe the actual system
Document purpose, users, inputs, models, tools, outputs, dependencies and affected people.
Deliverable: versioned system card.
- 02
Classify actions and data
Separate reading, recommending, drafting and executing. Identify prohibited, sensitive, personal and retained data.
Deliverable: data × action matrix.
- 03
Reduce permissions
Use dedicated accounts, tool allowlists, isolated environments and approval before high-impact action.
Deliverable: authorisation policy.
- 04
Build evaluations
Test normal cases, ambiguity, injection, missing data, out-of-scope requests and irreversible errors. Set thresholds before the pilot.
Deliverable: test set and results.
- 05
Monitor production
Log system version, necessary context, tool calls, approvals and outcomes while minimising retained data.
Deliverable: monitoring dashboard and alerts.
- 06
Prepare for failure
Define shutdown, manual takeover, notification, incident analysis and return-to-service criteria.
Deliverable: AI incident runbook.
From proposal to action: where are permissions checked?
An agent can prepare a request without permission to execute it. The executing service must check the actual identity, resource and parameters; a model claiming approval is not evidence of it.
Fictional workshop: an internal policy requires approval before sending a customer reply. These decisions illustrate that policy, without certifying or checking your system.
- Received data
A ticket or document supplies context, without granting new permissions.
- Proposed action
The agent prepares the tool, target and parameters for review.
- Independent check
The executing service verifies permissions and approval bound to this action.
- Observed outcome
A record links the unique request to its actual outcome, including failure.
Four cases to rehearse
The recipient changed
The approved preview named an internal address; the final call names an external one.
Expected decision
Hold the send and review the new destination.
Evidence to seek
Compare recipient, content and attachments in the preview and final call.
The wrong customer workspace
The account can read case A; the agent requests a file from case B.
Expected decision
Deny access to case B, even if the model says it needs it.
Evidence to seek
Test permission on the exact resource using the user's identity.
Approval has expired
The action matches the preview, but approval expired before execution.
Expected decision
Obtain fresh approval and recheck current permissions.
Evidence to seek
Retain expiry, approver identity and approved parameters.
The request timed out
The send request times out; whether it completed is unknown.
Expected decision
Check operation state before retrying.
Evidence to seek
Bind the idempotency key to the same content and reconcile the service receipt.
References : OWASP · LLM06:2025 Excessive Agency · OWASP · AI Agent Security Cheat Sheet
Worked example
Illustrative situation
An agent may read customer requests and draft a reply, but may not send a message or change an account without approval. Testing includes a source document that attempts to redirect its instructions.
Decision and expected evidence
The register defines permitted tools, visible data, escalation thresholds, necessary logs and the point at which a human takes over.
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Useful success rate | Tasks correct against a business criterion, not just plausible answers | Review failures by case family |
| Human escalation | Cases correctly handed to a person | Tune thresholds when the agent is overconfident |
| Cancelled actions | Actions blocked or recovered before impact | Find excessive permissions and untested scenarios |
| Drift | Performance difference by version, data or population | Compare against a stable set at every change |
Common pitfalls
- Measuring writing quality only
- Giving the agent an end user’s full rights
- Retaining every conversation by default
- Changing model or prompt without rerunning evaluations
Frequently asked questions
How is an AI agent different from a chatbot?
An agent plans or chains actions and may call tools. That power needs stricter permissions, approvals, traces and shutdown controls.
Is human approval always required?
It matters most when an action is irreversible, sensitive, regulated, financial or hard to detect afterwards. The level should follow risk.
What should be logged?
System version, strictly necessary context, tool calls, approvals, outcome and errors, with a justified retention period.
References: data, uses and limitations
Documentation reviewed on .
| Source and purpose | Verified data or benchmark | How to use it | Limitation to retain |
|---|---|---|---|
| MITRE ATLAS Examine documented tactics, techniques and cases involving AI systems. |
The official repository publishes ATLAS tactics, techniques and case-study data. They support analysis of the actual deployed system. | Select scenarios matching your application’s data, interfaces and tools. Test permissions and consequences, not only model responses. | A technique catalogue is neither an exhaustive coverage guarantee nor a test result for your system. |
| NIST · AI Risk Management Framework Organise management of AI-related risks. |
The AI RMF Core is organised around Govern, Map, Measure and Manage. It is a voluntary risk-management framework. | Define intended use, affected people and error consequences. Connect each risk to an evaluation, owner and stop or recovery mechanism. | The framework certifies neither a model nor an application. Using it does not remove the need to examine applicable requirements. |
| OWASP · LLM Top 10 Prepare tests for language-model application risks. |
The 2025 edition includes Prompt Injection, Sensitive Information Disclosure and Excessive Agency. | Turn a risk into an observable scenario: hostile document, restricted data or out-of-scope tool. Record expected behaviour and evidence of refusal. | A risk list is not a complete assessment. Input filtering alone does not establish authorisation for every action. |
RAG: access rights, citations and caches
Describe the authority of every agent tool
A tool’s authority depends on identity, objects and permitted operations. A document source, including an internal one, cannot expand that scope. The agent should accomplish the legitimate task with limited rights.
- Build an identity, object and operation matrix.
- Verify connector denials and absence of effects.
- Replay positive controls after each restriction.
Explore the RAG access matrix and replay cases →
References consulted on · OWASP · LLM06:2025 Excessive Agency OWASP · API1:2023 Broken Object Level Authorization NIST AI 100-2 E2025 · Adversarial Machine Learning
OAuth/OIDC: audience and access
An AI agent retains each tool’s authorisation boundaries
A model instruction does not create a grant. For every tool, identify the calling client, resource server, audience and object policy.
- Restrict permissions to the operation and recipient needed for the use case.
- Do not forward one tool’s token to another API merely for convenience.
- Test that an agent cannot read another tenant’s objects despite having a read scope.
See token recipients and the ten OAuth/OIDC cases →
Sources reviewed on · IETF · RFC 9700 · OAuth 2.0 Security BCP · IETF · RFC 9068 · JWT access token profile · OpenID Foundation · OpenID Connect Core 1.0 · IETF · RFC 8725 · JWT Best Current Practices · IETF · RFC 6749 · OAuth 2.0 framework
SSRF: URLs and connections
An AI agent tool keeps bounded network access
Text produced by a model, retrieved document or search result must not expand an allowed tool destination. Defining the connector by operation and service makes the agent’s actual network reach controllable.
- Reconstruct requests from business identifiers when the service is known.
- Enforce resolution, connection, proxy and each redirect at the egress component.
- Keep a traceable decision and response limits, including after retry or fallback.
Read the 12 cases and connection controls →
Sources reviewed on · OWASP · SSRF Prevention Cheat Sheet · WHATWG · URL Living Standard · AWS EC2 · Instance metadata options · IETF · RFC 3849
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
- NIST — AI Risk Management Framework
- NIST — Generative AI Profile
- OWASP · LLM06:2025 Excessive Agency
- OWASP · AI Agent Security Cheat Sheet
- MITRE ATLAS
- OWASP · LLM Top 10
- OWASP · API1:2023 Broken Object Level Authorization
- NIST AI 100-2 E2025 · Adversarial Machine Learning
- IETF · RFC 9700 · OAuth 2.0 Security BCP
- IETF · RFC 9068 · JWT access token profile
- OpenID Foundation · OpenID Connect Core 1.0
- IETF · RFC 8725 · JWT Best Current Practices
- IETF · RFC 6749 · OAuth 2.0 framework
- OWASP · SSRF Prevention Cheat Sheet
- WHATWG · URL Living Standard
- AWS EC2 · Instance metadata options
- IETF · RFC 3849
Continue exploring
Related analysis
