Resources · 09
RAG quality: evaluate a knowledge assistant with evidence
Measure corpus, retrieval, citations, answers, abstention and drift separately so that you know what genuinely needs fixing.
· 17 min
What this guide helps achieve
- Know whether failure comes from corpus, retrieval or model
- Trace every citation
- Reward appropriate abstention
- Detect drift after change
Quick check
- Is the reference source authorised and current?
- Does the right passage appear near the top?
- Is every decisive claim supported?
- Can the assistant say it does not know?
- Are sensitive cases escalated?
Step-by-step method
- 01
Define useful tasks
Collect real questions, their variants and the expected business result. Add ambiguity, source conflict, expired documents and out-of-scope requests.
Deliverable: case and risk taxonomy.
- 02
Clean the corpus
Assign an owner, rights, validity date and status to every source. Remove duplicates, orphan fragments and documents of unknown authority.
Deliverable: corpus register.
- 03
Evaluate retrieval
Measure whether necessary passages are found and ranked before assessing final wording. Inspect queries for which no evidence exists.
Deliverable: coverage and relevance measures.
- 04
Evaluate answers and citations
Check faithfulness to passages, completeness, limitations, the citation-to-claim relationship and the ability to flag contradiction.
Deliverable: faithfulness and utility rubric.
- 05
Test refusal and attacks
Add indirect injection, prohibited data requests, malicious sources, false priority documents and irreversible actions. Define human escalation.
Deliverable: security results and guardrails.
- 06
Monitor change
Version corpus, index, settings, model and instructions. Replay the stable set on every change and analyse real feedback without retaining unnecessary data.
Deliverable: drift dashboard and review procedure.
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Retrieval coverage | Questions for which necessary passages are found | Fix corpus, chunking or query before the model |
| Faithfulness | Claims genuinely supported by cited sources | Block or rewrite unsupported answers |
| Useful abstention | Cases without evidence correctly refused or escalated | Tune confidence thresholds |
| Regression | Quality difference between versions on the same set | Identify the responsible change |
Common pitfalls
- Measuring writing fluency only
- Mixing retrieval failure and hallucination
- Using an evaluation set that is too easy
- Changing several components without comparison
Frequently asked questions
How many questions are needed?
Start with a representative, versioned set covering important risks. Diversity and quality matter more than artificial volume.
Is one automated metric enough?
No. It accelerates triage, but sensitive, ambiguous and domain cases need explicit criteria and human review.
Should every conversation be retained?
No. Keep only necessary traces with a defined purpose, access and duration. Prefer anonymised cases for improvement.
