Resources · 03
AI agent governance: a production readiness guide
Frame permissions, data, tests, oversight and shutdown before an agent is trusted with a business process.
· 2 min
What this guide helps achieve
- Limit the agent to what it needs
- Test costly failures before production
- Keep useful traces without over-collecting
- Maintain effective human accountability
Quick check
- Can the agent trigger an irreversible action?
- Is the submitted data authorised?
- Does a threshold require human approval?
- Do tools have minimum permissions?
- Can the system stop and roll back quickly?
Step-by-step method
- 01
Describe the actual system
Document purpose, users, inputs, models, tools, outputs, dependencies and affected people.
Deliverable: versioned system card.
- 02
Classify actions and data
Separate reading, recommending, drafting and executing. Identify prohibited, sensitive, personal and retained data.
Deliverable: data × action matrix.
- 03
Reduce permissions
Use dedicated accounts, tool allowlists, isolated environments and approval before high-impact action.
Deliverable: authorisation policy.
- 04
Build evaluations
Test normal cases, ambiguity, injection, missing data, out-of-scope requests and irreversible errors. Set thresholds before the pilot.
Deliverable: test set and results.
- 05
Monitor production
Log system version, necessary context, tool calls, approvals and outcomes while minimising retained data.
Deliverable: monitoring dashboard and alerts.
- 06
Prepare for failure
Define shutdown, manual takeover, notification, incident analysis and return-to-service criteria.
Deliverable: AI incident runbook.
Your acceptance workbook
Record observations against this guide’s criteria. A record is not certification.
The workbook does not save automatically. Export before leaving.
Filters do not limit exports. Actions include issues and unreviewed criteria.
Import replaces current observations after your confirmation.
Worked example
Illustrative situation
An agent may read customer requests and draft a reply, but may not send a message or change an account without approval. Testing includes a source document that attempts to redirect its instructions.
Decision and expected evidence
The register defines permitted tools, visible data, escalation thresholds, necessary logs and the point at which a human takes over.
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Useful success rate | Tasks correct against a business criterion, not just plausible answers | Review failures by case family |
| Human escalation | Cases correctly handed to a person | Tune thresholds when the agent is overconfident |
| Cancelled actions | Actions blocked or recovered before impact | Find excessive permissions and untested scenarios |
| Drift | Performance difference by version, data or population | Compare against a stable set at every change |
Common pitfalls
- Measuring writing quality only
- Giving the agent an end user’s full rights
- Retaining every conversation by default
- Changing model or prompt without rerunning evaluations
Frequently asked questions
How is an AI agent different from a chatbot?
An agent plans or chains actions and may call tools. That power needs stricter permissions, approvals, traces and shutdown controls.
Is human approval always required?
It matters most when an action is irreversible, sensitive, regulated, financial or hard to detect afterwards. The level should follow risk.
What should be logged?
System version, strictly necessary context, tool calls, approvals, outcome and errors, with a justified retention period.
Official references
Continue exploring
Related analysis
