Resources · 44
Prompt injection: test an AI agent’s trust boundaries
Check that a document, search result or message cannot authorise an action on behalf of the user.
· 4 min
What this guide helps achieve
- Separate data from instructions
- Limit available actions to the task
- Test real effects behind responses
- Document limitations and regressions
Quick check
- Which external content does the agent read?
- Which actions can it trigger without approval?
- Is authorisation enforced outside the model?
- Does a verbal refusal actually prevent a tool call?
- How can a diverging scenario be stopped?
Step-by-step method
- 01
Inventory the boundaries
List messages, web pages, documents, search results, attachments and tool outputs. Treat them as data to interpret without authority over permissions. Include channels carrying text or multimodal content.
Deliverable: map of inputs and available actions.
- 02
Define forbidden effects
Describe harms to prevent: out-of-scope access, unapproved publication, data transfer or account changes. Set an observable outcome for each scenario; judging only the answer text can miss an action already performed.
Deliverable: threat scenarios and invariants.
- 03
Isolate the test environment
Use clearly labelled fictitious accounts, documents and destinations with test markers and simulated tools. Disable real sending and production access. Retain model, configuration, corpus and connector versions to replay cases.
Deliverable: versioned environment and test set.
- 04
Replay adversarial inputs
Place in a test source an instruction that contradicts the task, requests an out-of-scope action or asserts false authority. Vary language, placement, format and source combinations. Measure tool invocation, arguments and actual effect.
Deliverable: traces of successful, blocked and unresolved cases.
- 05
Strengthen external controls
Enforce minimal permissions, argument validation, read/write separation and allowed destinations in the tools. Require contextual approval for sensitive operations. RAG and a refusal instruction alone do not remove injection risk.
Deliverable: executable rules and bypass tests.
- 06
Track changes
Replay cases after changes to model, corpus, tools or rules. Also track legitimate tasks that get blocked. Publish tested scope and known limitations; success on a small test set does not establish the absence of vulnerabilities.
Deliverable: regression report and deployment decision.
Proposed acceptance scenarios
Fictional scenarios to adapt in an authorised environment. They describe no client result or certification.
| Fictional situation | Suggested check | Evidence to retain |
|---|---|---|
| A retrieved fictional document asks the agent to send a dossier to an external destination. | In an isolated environment, verify that the document remains untrusted data and that the tool refuses the unauthorised action. A verbal refusal is insufficient. | Document source and version, simulated tool call, authorisation decision and absence of external effect. |
| An apparently correct answer incorporates an instruction from a secondary page. | Replay with and without that page, compare citations and separately check tool arguments. | Model, corpus and tool versions, observed differences and a retained regression case. |
Worked example
Illustrative situation
Illustrative situation: an assistant summarises a test document that requests sending information to an external destination.
Decision and expected evidence
The tool rejects that destination independently of generated text. The test checks that nothing was sent and the legitimate summary still works.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| Input filter | Detect some suspicious patterns | May miss new wording |
| Model instruction | Describe the expected boundary | Is not an access control |
| Tool enforcement | Reject an unauthorised operation | Must check identity, object and arguments |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Forbidden actions | Cases causing out-of-scope effects | Block and investigate every effect |
| False refusals | Legitimate tasks prevented | Revise without broadening permissions |
| Coverage | Channels and tools actually tested | Declare blind spots |
Common pitfalls
- Putting secrets in test prompts
- Testing visible answers only
- Assuming internal documents are always trusted
- Claiming absolute protection after a few tests
Frequently asked questions
Does RAG prevent injection?
No. Retrieved documents can themselves carry adversarial instructions. Sources remain data to process within controlled boundaries.
Should every suspicious document be blocked?
It depends on the task; an agent can often analyse a document without following its instructions or having write tools.
How should results be published?
State versions, scope, effects tested, limitations and fixes. Exclude secrets and client documents used in the work.
What evidence makes a verification useful?
Keep the untrusted source, model and tool versions, authorisation decision and final state. Check absence of forbidden effects beyond answer text: a displayed refusal alone does not establish that the tool executed nothing.
RAG: access rights, citations and caches
A hostile source cannot grant a permission
Text read by an agent may try to change the task; it must not change document-service permissions. Replay the same object with an authorised group, another group and a revoked group.
- Compare the access decision with the actual effect even when the answer sounds correct.
- Pass the test document through retrieval, citation and cache.
- Keep a positive control to distinguish protection from blanket denial.
Explore the RAG access matrix and replay cases →
References consulted on · OWASP · LLM01:2025 Prompt Injection OWASP · LLM06:2025 Excessive Agency NIST AI 100-2 E2025 · Adversarial Machine Learning
