Deploy useful AI with appropriate controls

Lista de ações

1. 01

   Define the task and boundaries

   Describe inputs, expected outputs, users and decisions the AI must not make. Compare the expected benefit with a simpler solution without a generative model. Example: retrieve the internal procedure applicable to a request, including the cited passage and version date. Success requires a correct source accessible to the requester. A plausible answer without evidence should fail even when it reads convincingly.

2. 02

   Map the data

   Document the data sent, its location, retention and provider terms. Minimise sensitive information and check permissions on indexed documents. For RAG, distinguish permission to index, retrieve and disclose. A document accessible to an administrator must not become accessible to everyone. Test the same topic with different roles and preserve expected refusals.

3. 03

   Evaluate representative cases

   Build normal, difficult and adversarial cases with explicit pass criteria. Measure factual errors, citations, refusals, cost and latency, and retain the tested versions. Keep a control set separate from examples used to tune instructions. Have a competent reviewer annotate responses and track incorrect citations, incomplete answers and justified refusals separately. Compare results with straightforward document search.

4. 04

   Constrain agent capabilities

   Grant minimum permissions. Require human approval for consequential actions, separate environments and provide a stop mechanism and rollback. A correct answer does not authorise sending a message, changing an account or deleting a document. Constrain tools by action and resource; require explicit confirmation when an operation commits its recipient or changes sensitive data.

5. 05

   Test hostile inputs

   Treat retrieved documents and pages as untrusted data. Test prompt injection, exfiltration, tool misuse and unsupported answers; do not assume filtering solves every issue. Put a conflicting instruction in a test document and observe whether the system treats it as content. Also verify that a tool result cannot become an authorisation. Test the complete workflow rather than the model alone.

6. 06

   Monitor and reassess

   Assign an owner, monitor incidents and repeat evaluations when models, prompts or the corpus change. Define a stop threshold and a manual alternative. Retain proportionate logs that identify the case, sources and human decision without unnecessarily duplicating sensitive data. Replay critical cases after changes and provide a documented manual fallback.

   Worked example: an assistant answers contract questions. Test the same file with an authorised role and an excluded role. The first should receive an answer grounded in the right passage; the second must not reveal restricted content or excerpts. Add an unrelated document instructing the assistant to email a file: inclusion in the corpus must not authorise that action.