Resources · 03
AI agent governance: a production readiness guide
Frame permissions, data, tests, oversight and shutdown before an agent is trusted with a business process.
· 15 min
What this guide helps achieve
- Limit the agent to what it needs
- Test costly failures before production
- Keep useful traces without over-collecting
- Maintain effective human accountability
Quick check
- Can the agent trigger an irreversible action?
- Is the submitted data authorised?
- Does a threshold require human approval?
- Do tools have minimum permissions?
- Can the system stop and roll back quickly?
Step-by-step method
- 01
Describe the actual system
Document purpose, users, inputs, models, tools, outputs, dependencies and affected people.
Deliverable: versioned system card.
- 02
Classify actions and data
Separate reading, recommending, drafting and executing. Identify prohibited, sensitive, personal and retained data.
Deliverable: data × action matrix.
- 03
Reduce permissions
Use dedicated accounts, tool allowlists, isolated environments and approval before high-impact action.
Deliverable: authorisation policy.
- 04
Build evaluations
Test normal cases, ambiguity, injection, missing data, out-of-scope requests and irreversible errors. Set thresholds before the pilot.
Deliverable: test set and results.
- 05
Monitor production
Log system version, necessary context, tool calls, approvals and outcomes while minimising retained data.
Deliverable: monitoring dashboard and alerts.
- 06
Prepare for failure
Define shutdown, manual takeover, notification, incident analysis and return-to-service criteria.
Deliverable: AI incident runbook.
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Useful success rate | Tasks correct against a business criterion, not just plausible answers | Review failures by case family |
| Human escalation | Cases correctly handed to a person | Tune thresholds when the agent is overconfident |
| Cancelled actions | Actions blocked or recovered before impact | Find excessive permissions and untested scenarios |
| Drift | Performance difference by version, data or population | Compare against a stable set at every change |
Common pitfalls
- Measuring writing quality only
- Giving the agent an end user’s full rights
- Retaining every conversation by default
- Changing model or prompt without rerunning evaluations
Frequently asked questions
How is an AI agent different from a chatbot?
An agent plans or chains actions and may call tools. That power needs stricter permissions, approvals, traces and shutdown controls.
Is human approval always required?
It matters most when an action is irreversible, sensitive, regulated, financial or hard to detect afterwards. The level should follow risk.
What should be logged?
System version, strictly necessary context, tool calls, approvals, outcome and errors, with a justified retention period.