Possible deliverables
Possible deliverables
- AI opportunity map
- Assistant, RAG or agent prototype
- Test set and quality protocol
- Usage, privacy and governance framework
03 / 06
Cyber · AI · Digital intelligence
Applied AI & agents
Turn a business use case into a useful, controllable AI system compatible with your confidentiality requirements.
We start from real work: available information, decisions to support, repetitive tasks, acceptable error and sensitive data. Prototyping follows framing, with explicit verification and human accountability.
03
What the assignment should give you
- 01Use cases ranked by value and feasibility
- 02A measurable prototype rather than a vague demo
- 03Clear rules for data, control and limitations
How the assignment works
How the assignment works
A short, explicit process adapted to the sensitivity of the situation.
- 01
Scoping
We define the decision to support, relevant assets, constraints, authorised sources and required confidentiality.
- 02
Investigation
Collection and analysis follow the agreed scope. Facts, hypotheses and areas of uncertainty remain clearly separated.
- 03
Prioritisation
Each finding is assessed by reliability, likely impact and the effort required to act.
- 04
Delivery
You receive a readable synthesis, useful evidence and a roadmap that can be put into practice.
Common situations
Common situations
- Search across an internal knowledge base
- Request qualification or case synthesis
- Controlled automation of repetitive work
How can the assignment’s result be verified?
Use these examples to agree on a verifiable deliverable. Checks and thresholds follow the agreed scope; the examples below are not client results.
| What to check | Evidence to request | Decision it supports |
|---|---|---|
| Evaluation set | Representative and ambiguous cases, reference results, exclusions and dataset version; cases used for tuning kept separate. | Assess the intended use without confusing a demonstration with an evaluation. |
| Detailed results | Case counts by type, successes, critical errors, expected refusals and human review of disagreements. | Set limits by risk; a favourable average does not cancel a serious error. |
| Action permissions | Tool/action/data matrix, access controls, approval requests and refusal tests. | Permit only intended actions and block actions outside the mandate. |
| Operations | Versioned model, instructions and configuration; cost, latency, drift, stop threshold and human handover procedure. | Maintain, fix or suspend the system when conditions change. |
Put the decision to the test
96 correct answers: can actions be enabled?
An assistant is evaluated on one hundred service requests. Its overall score looks favourable, but one error concerns an action that changes an account.
Fictional teaching case. Figures explain the reasoning; they are neither client results nor thresholds required by a standard.
- requests evaluated
- 100
- answers judged correct
- 96
- unauthorised account change among the four errors
- 1
| Possible choice | Missing evidence or supporting reason |
|---|---|
| Enable all actions at 96% success | The average hides error severity and type. |
| Abandon every use of the system | Classification or drafting can be assessed separately from modification rights. |
| Keep read-only mode and block changes | Fix the authorisation control before reassessing the affected action. |
Read the decision and next check
The score describes 96 correct cases on this particular set. It is insufficient to permit account changes. Read-only use can have a separate pilot, with review of the other three errors and explicit suspension conditions.
The next check
Replay the same case set with an outdated document, revoked access and an unanswerable question. Check refusal, citations and absence of unauthorised action. A better average cannot compensate for a data leak.
What this case does not demonstrate
One hundred cases cannot cover every future situation. An observed correct answer does not prove that permissions are enforced.
References and scope
- NIST — AI RMF Playbook, Measure
MEASURE 2.1 calls for documenting test sets, metrics and tools; MEASURE 2.3 connects evaluation with deployment conditions. The framework informs control selection without prescribing a universal reliability threshold.
- NIST — AI RMF 1.0: Core
Govern, Map, Measure and Manage organise continuous risk management. The framework supplies no success rate that automatically authorises an action.
References checked on 15 October 2026.
Before the assignment
Frame a useful decision
The starting point is a task, its volume and the cost of a wrong answer or action. Scope defines accessible data, tool permissions and moments when a person must approve or take over. Scoping distinguishes text generation, RAG retrieval and tool execution. Evaluation preserves model, instruction and document versions, checking citations, refusals, access control and errors with business consequences. Cost per successfully completed task and human recovery time complement quality measures. Deployment depends on criteria agreed before testing.
Information to gather
- Real examples of successful and failed tasks
- Documents, access rights and retention constraints
- Quality criteria and acceptable error costs
Decisions to prepare
- Is a simple assistant, document search or agent needed?
- Which actions require approval?
- Which tests block release?
Illustrative assignment
- Situation
- A service wants to automate request triage while retaining approval for replies and account changes.
- Work
- Compare a human-labelled baseline with the assistant on representative, ambiguous and adversarial cases; restrict tools and trace refusals.
- Deliverable
- A versioned test protocol, permission matrix, human approval thresholds and production stop criteria.
Example acceptance criteria to agree during scoping
- Useful and forbidden tasks have distinct test cases.
- An untrusted source cannot broaden tool permissions.
- Shutdown criteria and human handoff are exercised.
Related public methods
References: data, uses and limitations
Documentation reviewed on .
| Source and purpose | Verified data or benchmark | How to use it | Limitation to retain |
|---|---|---|---|
| MITRE ATLAS Examine documented tactics, techniques and cases involving AI systems. |
The official repository publishes ATLAS tactics, techniques and case-study data. They support analysis of the actual deployed system. | Select scenarios matching your application’s data, interfaces and tools. Test permissions and consequences, not only model responses. | A technique catalogue is neither an exhaustive coverage guarantee nor a test result for your system. |
| NIST · AI Risk Management Framework Organise management of AI-related risks. |
The AI RMF Core is organised around Govern, Map, Measure and Manage. It is a voluntary risk-management framework. | Define intended use, affected people and error consequences. Connect each risk to an evaluation, owner and stop or recovery mechanism. | The framework certifies neither a model nor an application. Using it does not remove the need to examine applicable requirements. |
| OWASP · LLM Top 10 Prepare tests for language-model application risks. |
The 2025 edition includes Prompt Injection, Sensitive Information Disclosure and Excessive Agency. | Turn a risk into an observable scenario: hostile document, restricted data or out-of-scope tool. Record expected behaviour and evidence of refusal. | A risk list is not a complete assessment. Input filtering alone does not establish authorisation for every action. |
RAG: access rights, citations and caches
Define acceptance criteria for a document assistant
A pilot should demonstrate useful access for authorised groups and observable denial elsewhere. A separate index or confidentiality instruction does not validate the entire journey.
- Define sentinel documents, identities and expected outcomes.
- Measure revocation propagation across every path.
- Deliver discrepancies with evidence, owner and conditions for resumption.
Explore the RAG access matrix and replay cases →
References consulted on · OWASP · LLM06:2025 Excessive Agency OWASP · API1:2023 Broken Object Level Authorization NIST AI 100-2 E2025 · Adversarial Machine Learning
OAuth/OIDC: audience and access
An AI agent retains each tool’s authorisation boundaries
A model instruction does not create a grant. For every tool, identify the calling client, resource server, audience and object policy.
- Restrict permissions to the operation and recipient needed for the use case.
- Do not forward one tool’s token to another API merely for convenience.
- Test that an agent cannot read another tenant’s objects despite having a read scope.
See token recipients and the ten OAuth/OIDC cases →
Sources reviewed on · IETF · RFC 9700 · OAuth 2.0 Security BCP · IETF · RFC 9068 · JWT access token profile · OpenID Foundation · OpenID Connect Core 1.0 · IETF · RFC 8725 · JWT Best Current Practices · IETF · RFC 6749 · OAuth 2.0 framework
SSRF: URLs and connections
An AI agent tool keeps bounded network access
Text produced by a model, retrieved document or search result must not expand an allowed tool destination. Defining the connector by operation and service makes the agent’s actual network reach controllable.
- Reconstruct requests from business identifiers when the service is known.
- Enforce resolution, connection, proxy and each redirect at the egress component.
- Keep a traceable decision and response limits, including after retry or fallback.
Read the 12 cases and connection controls →
Sources reviewed on · OWASP · SSRF Prevention Cheat Sheet · WHATWG · URL Living Standard · AWS EC2 · Instance metadata options · IETF · RFC 3849
AI: check a JSON result
An extraction contract precedes every agent action
An AI-generated object remains an input to validate. Correct types establish neither document values, actor identity nor permission to write into the target system.
- Validate fields before handing them to the connector.
- Route unknown values and missing evidence into an explicit review path.
- Apply context and authorization controls to every downstream operation.
Compare the 14 outputs and their reasons →
Sources reviewed on · IETF · RFC 8259 · JSON Schema · Object reference · JSON Schema · Draft 2020-12 validation · Python 3.12 · json decoder · python-jsonschema · Schema validation · OWASP · LLM05:2025 Improper Output Handling
RAG relevance and ranking
Compare retrieval before evaluating the answer
Keep the same needs, judgments and depth k when comparing configurations. Inspect regressing queries before drawing conclusions from an average.
- Separate window precision, relevant-record coverage and ordering quality.
- Document missing judgments and the no-relevant-record convention.
- Then evaluate the answer, citations and refusals using a separate protocol.
Explore ranking measures and relevance judgments →
Stanford CS276 · Retrieval evaluation · 2019 course archive · PDF pp. 3, 5–6 · consulted 2026-10-23
NIST TREC · Reciprocal rank definition · Repository snapshot 35cf096ba6d5ae08ca4803c056b31ed61a56a6d4 · consulted 2026-10-23
FAQ
Frequently asked questions
Practical answers to help frame the need before an initial conversation.
Do we need to choose an AI model or platform first?
No. The technical choice follows analysis of the need, available data, acceptable error, confidentiality constraints and operating cost.
How do you measure the reliability of an assistant or agent?
Representative test cases measure accuracy, traceability, expected refusals and critical errors. Results define human controls and appropriate limits of use.
What needs retesting after an AI assistant changes?
Replay a stable set of authorised, ambiguous and prohibited requests after a model, instruction, corpus or tool change. Compare answers, actions and refusals by case type, recording version and permissions. Earlier acceptance does not automatically cover new behaviour.
Digital ecosystem
Extend the analysis with specialist resources.
Française du Numérique
Web creation, SEO and content
francaise-du-numerique.fr
ENExplore
French Digital Society
Digital professions and expertise
french-digital-society.com
ENExplore
Internet Intelligence Service
OSINT, monitoring and due diligence
internet-intelligence-service.com
ENExplore
CFSN
Digital support for small businesses
cfsn.fr
French editionExplore
World Affiliate Guide
Affiliate and performance marketing
world-affiliate-guide.com
ENExplore
World Social Networks
Social networks, usage and privacy
world-social-networks.com
ENExplore
World Web Certified
Curated web directory and guides
worldwebcertified.com
ENExplore