Resources · 09

RAG quality: evaluate a knowledge assistant with evidence

Measure corpus, retrieval, citations, answers, abstention and drift separately so that you know what genuinely needs fixing.

Updated 23 October 2026 · 11 min

Illustration · fictional scene

What this guide helps achieve

Know whether failure comes from corpus, retrieval or model
Trace every citation
Reward appropriate abstention
Detect drift after change

Quick check

Is the reference source authorised and current?
Does the right passage appear near the top?
Is every decisive claim supported?
Can the assistant say it does not know?
Are sensitive cases escalated?

Step-by-step method

01
Define useful tasks
Collect real questions, their variants and the expected business result. Add ambiguity, source conflict, expired documents and out-of-scope requests.
Deliverable: case and risk taxonomy.
02
Clean the corpus
Assign an owner, rights, validity date and status to every source. Remove duplicates, orphan fragments and documents of unknown authority.
Deliverable: corpus register.
03
Evaluate retrieval
Measure whether necessary passages are found and ranked before assessing final wording. Inspect queries for which no evidence exists.
Deliverable: coverage and relevance measures.
04
Evaluate answers and citations
Check faithfulness to passages, completeness, limitations, the citation-to-claim relationship and the ability to flag contradiction.
Deliverable: faithfulness and utility rubric.
05
Test refusal and attacks
Add indirect injection, prohibited data requests, malicious sources, false priority documents and irreversible actions. Define human escalation.
Deliverable: security results and guardrails.
06
Monitor change
Version corpus, index, settings, model and instructions. Replay the stable set on every change and analyse real feedback without retaining unnecessary data.
Deliverable: drift dashboard and review procedure.

Understand · try · decide

AI answers: make abstentions and denominators visible

The NIST generative profile addresses confabulation and context-dependent evaluation. For a document assistant, we propose separately counting tested questions, produced answers and answers validated against documents. These formulas are a CIE teaching grid, with no universal threshold.

Define “validated” before testing: an accurate answer, a citation that actually supports the claim, an authorised source and sufficient freshness for the question. A link in the text does not automatically meet these criteria. Record model, corpus and instruction versions.

Separate ordinary questions, missing documents, conflicting sources and out-of-scope requests. Expected abstention on an unanswerable question can be correct. The simulator describes only the subset where an answer was expected; expected abstentions and critical errors need separate results.

Reference points and evidence to retain

Reference point	What you can conclude	Evidence to retain
Tested questions	Scope where an answer was expected	Question set and evaluation rule
Produced answers	Share of questions receiving an answer	Retained answers and abstentions
Validated answers	Answers meeting the declared criterion	Verdict and supporting document passage
Unanswered questions	Tested questions − produced answers	Reason and review of the abstention

From observation to decision

Define the expected truth
Freeze corpus and version
Check every answer
Compare by question family

Break down a result before deciding

The same hundred-question set can achieve 90% quality among answers and only 72% success across the full set. This breakdown uses the fictional example, with an answer expected for each question. It is neither a vendor benchmark nor a NIST score.

Validated answers: 72 / 10072% of the set
Unvalidated answers: 8 / 1008% of the set
Unanswered questions: 20 / 10020% of the set

Where do the 100 questions go?

72 / 100 · Validated
8 / 100 · Unvalidated
20 / 100 · Unanswered

100 = 72 + 8 + 20. Of the 80 produced answers, 72 are validated: 90%. The twenty unanswered questions remain visible.

A global average can hide a weak question family
Fictional family	Tested / produced / validated	Interpretation
Ordinary questions	80 / 78 / 72	Quality 72/78 ≈ 92.31%; coverage 78/80 = 97.5%.
Sensitive questions	20 / 2 / 0	Quality 0/2 = 0%; coverage 2/20 = 10%.
Combined	100 / 80 / 72	Quality 90%; coverage 80%; success 72%.

These counts illustrate differences between subgroups. Small samples do not justify generalisation to all future questions.

Checks before concluding

Record factual error, unsupported citation, stale source, unauthorised access and abstention separately. An average does not describe severity.
For a cited passage, verify evidence is present and supports the claim. Also test removed documents and documents accessible to another role.
Keep one set for expected answers and another for expected abstentions; declare both denominators and evaluation rules.
After corpus, model or permission changes, replay the same cases with recorded versions. Address critical errors before setting a business threshold.

Decision: investigate the sensitive family and the eight unvalidated answers. The global 90% result does not authorise a production release.

Calculation rule

Quality = validated / produced. Coverage = produced / tested. Success across the set = validated / tested. A zero denominator yields a result that cannot be calculated.

Change an assumption

Worked example

Illustrative situation

An internal assistant answers from procedures, some of which have expired. The test set contains routine, ambiguous and unanswerable questions; each citation is checked against the document version.

Decision and expected evidence

Release waits until expired sources are excluded, unsupported answers are refused and the corpus owner is identified.

Management indicators

Indicator	What it measures	First action
Retrieval coverage	Questions for which necessary passages are found	Fix corpus, chunking or query before the model
Faithfulness	Claims genuinely supported by cited sources	Block or rewrite unsupported answers
Useful abstention	Cases without evidence correctly refused or escalated	Tune confidence thresholds
Regression	Quality difference between versions on the same set	Identify the responsible change

Common pitfalls

Measuring writing fluency only
Mixing retrieval failure and hallucination
Using an evaluation set that is too easy
Changing several components without comparison

Frequently asked questions

How many questions are needed?

Start with a representative, versioned set covering important risks. Diversity and quality matter more than artificial volume.

Is one automated metric enough?

No. It accelerates triage, but sensitive, ambiguous and domain cases need explicit criteria and human review.

Should every conversation be retained?

No. Keep only necessary traces with a defined purpose, access and duration. Prefer anonymised cases for improvement.

RAG: access rights, citations and caches

Evaluate an answer alongside its document rights

A relevant citation can support an accurate answer while revealing a forbidden document. RAG evaluation should separate faithfulness, freshness and authorisation of passages actually supplied to the model.

Annotate permitted documents for each test identity.
Remove a group and replay an existing conversation and citation.
Track incorrect answers, missing evidence and out-of-scope exposure separately.

Explore the RAG access matrix and replay cases →

References consulted on 2026-10-22 · OWASP · LLM08:2025 Vector and Embedding Weaknesses OWASP · API1:2023 Broken Object Level Authorization

AI: check a JSON result

A retrieved reference does not prove the extracted value

A RAG system can find the right document and return schema-valid JSON that misstates a field. Separate retrieval, the output contract and passage reconciliation to locate what needs correction.

Keep an independent reference for decisive fields.
Test missing values, contradictions and unavailable documents.
Retain unexecuted checks as such instead of counting them as passes.

Compare the 14 outputs and their reasons →

Sources reviewed on 2026-10-22 · IETF · RFC 8259 · JSON Schema · Object reference · JSON Schema · Draft 2020-12 validation · Python 3.12 · json decoder · python-jsonschema · Schema validation · OWASP · LLM05:2025 Improper Output Handling

RAG relevance and ranking

A RAG system can retrieve useful documents and still produce an incorrect answer. Retrieval evaluation therefore needs its own questions, relevance judgments and defined reading depth. This dossier explains four ranking measures without confusing them with answer faithfulness, access permissions or guaranteed business outcomes.

Documentary references: Stanford CS276’s evaluation material from the 2019 archive and NIST TREC’s reciprocal-rank definition. The CIE workshop uses binary judgments and one query at a time. RR@k here is truncated at k; MRR is a mean RR across multiple queries and is not computed by this workshop.

Stanford CS276 · Retrieval evaluation · 2019 course archive · PDF pp. 3, 5–6 · consulted 2026-10-23

Four measures, four different questions

Four measures, four different questions
Measure	Workshop calculation	Useful reading	Condition or limitation
Precision@k	Relevant results in the first k / k	Measures the useful proportion in the inspected window.	Unavailable if any judgment is missing in the first k.
Recall@k	Relevant results in the first k / R	R is the number of relevant records in the entire small judged corpus.	If R is unknown, publish a value only when it is defined and identical in every possible case.
RR@k	1 / rank of the first relevant result within k	Equals 0 when no relevant record appears in that fully judged window.	An unjudged record before the first relevant result makes that rank uncertain.
nDCG@k	DCG@k / IDCG@k	Compares ranking gain with the ideal order at the same k.	Unavailable if the value changes across possible cases or any case has a zero ideal.

Explicit convention: judgment rᵢ is 0 or 1; DCG@k = Σᵢ₌₁ᵏ rᵢ / log₂(i + 1). With binary relevance, gain 2ʳⁱ − 1 also equals rᵢ. IDCG@k puts min(R, k) relevant records first using the same discount. nDCG@k = DCG@k / IDCG@k when that ideal is defined and nonzero. Keep gain, discount, depth and conventions with any published score.

NIST TREC · Reciprocal rank definition · Repository snapshot 35cf096ba6d5ae08ca4803c056b31ed61a56a6d4 · consulted 2026-10-23

A fictional six-record corpus with declared judgments

Titles and judgments below are created for this exercise. They are neither real search results nor automatic title-based judgments. 1 means relevant, 0 nonrelevant and ? unjudged. Columns Q1–Q5 correspond to the needs displayed in the workshop.

Q1 · Prepare a restoration test
Q2 · Authorise a document read
Q3 · Check a quantum sensor
Q4 · Restoration test: C judgment missing
Q5 · Restoration: B unjudged, other records off-topic

A fictional six-record corpus with declared judgments
Fictional record	Q1	Q2	Q4	Q5
A · Restoration procedure	1	0	1	1
B · Recovery budget	1	0	1	?
C · Restoration test log	1	0	?	0
D · Account and permission guide	0	1	0	0
E · Archiving policy	0	0	0	0
F · Publication schedule	0	0	0	0

Document ranking workshop

Choose a need and enter an ordered list of A–F identifiers separated by commas. Change k to inspect only the first k results. The list may omit documents, but each identifier must be unique and k cannot exceed its length. Calculation stays local without collection, import, transmission, storage or export.

This reader queries no search engine, private corpus or AI model. It measures neither answer truth, correct citation nor permission security. Scores concern declared judgments in this small fictional universe. Do not present an exercise value as a benchmark of your system or a universal deployment threshold.

Fictional scenario

Information need

A–F, no duplicates, separated by commas; ASCII spaces allowed. k is an integer from 1 to 6, no greater than list length.

Ranking, identifiers A to F

Depth k

Result

Unjudged and nonrelevant are different states ↓

Calculation over the first k results
Rank	Fictional record	Judgment	Discount	DCG contribution

Unjudged and nonrelevant are different states

Unjudged and nonrelevant are different states
Situation	CIE educational convention
Missing judgment within k	Precision may vary. Reciprocal rank remains known if the first relevant record is identified before any unjudged record.
Missing judgment elsewhere	R becomes a range of possibilities. Do not assume recall and nDCG are always unknown: a value may remain constant, for example when the order is ideal in every case.
No relevant record in a fully judged corpus	Precision@k and RR@k are 0. The reader shows Recall@k and nDCG@k as unavailable for this zero-denominator case.
CIE publication rule	Explore every binary assignment of missing judgments, at most 64 for six records. Publish a measure only if it is defined and identical in every assignment. These are possibilities, with no probabilities assigned and no actual imputation.

Ten explained scenarios, available without JavaScript

K01 · Relevant records first

Relevant records first
Need	Supplied order	k	Expected reading
Q1	A,B,C,D,E,F	3	Precision@3 = 1, Recall@3 = 1, RR@3 = 1 and nDCG@3 = 1 in this fully judged corpus.

K02 · Relevant records after cutoff

Relevant records after cutoff
Need	Supplied order	k	Expected reading
Q1	D,E,F,A,B,C	3	All four measures are 0 at k = 3. Relevant records exist but are placed after the inspected window.

K03 · First relevant record at rank two

First relevant record at rank two
Need	Supplied order	k	Expected reading
Q1	D,A,E,B,F,C	3	Precision@3 and Recall@3 are 1/3; RR@3 is 1/2. nDCG considers position and the ideal order.

K04 · Relevant records omitted

Relevant records omitted
Need	Supplied order	k	Expected reading
Q1	A,D,E,F	3	R remains 3 in the corpus even though B and C are absent from the list. Recall must not use only returned results as its denominator.

K05 · Duplicate identifier

Duplicate identifier
Need	Supplied order	k	Expected reading
Q1	A,A,B	3	Input rejected: A must not be counted twice. Correct the ranking rather than inflating the relevant count.

K06 · Unjudged before the first relevant record

Unjudged before the first relevant record
Need	Supplied order	k	Expected reading
Q4	C,A,D,E,B,F	3	C is unjudged. Precision@3, Recall@3, RR@3 and nDCG@3 are unavailable under the declared conventions.

K07 · Unjudged after the window

Unjudged after the window
Need	Supplied order	k	Expected reading
Q4	A,B,D,E,F,C	3	Precision@3 is 2/3 and RR@3 is 1. Recall@3 and nDCG@3 remain unavailable because C’s corpus judgment is missing.

K08 · Need with no relevant record

Need with no relevant record
Need	Supplied order	k	Expected reading
Q3	A,B,C,D,E,F	3	Precision@3 and RR@3 are 0. Recall and nDCG remain unavailable; this is an explicit convention rather than division by zero.

K09 · Change reading depth

Change reading depth
Need	Supplied order	k	Expected reading
Q2	A,D,B,C,E,F	1	At k = 1 no relevant record is visible. At k = 2, D appears: precision 1/2, recall 1 and RR@2 = 1/2.

K10 · Unknown relevance, always ideal ordering

Unknown relevance, always ideal ordering
Need	Supplied order	k	Expected reading
Q5	A,B,C,D,E,F	2	B may be relevant or not. Either way, all relevant records are within the first two ranks and the order stays ideal: recall, RR and nDCG are 1. Precision remains unknown.

Prepare a reproducible evaluation

Prepare a reproducible evaluation
Suggested check	Record to retain
Needs	Representative and unanswerable questions; specify the documentary need behind each wording.
Corpus	Document and index versions, accessible scope and deletion or revocation rules.
Judgments	Relevance rule, assessors, disagreements and unjudged state; do not equate clicks with documentary truth.
Protocol	Same k, filters, data and conventions before and after a retrieval change.
Regression	Per-query and per-need-family results, gaining and losing cases, hypothesis and checks on new documents.
Final answer	Evaluate faithfulness, citations, useful refusals and permissions separately; better retrieval scores do not validate this stage.

For a reviewable decision, keep the question, ranking, judgments, k and corpus limitations with the score. One average can conceal a regression on a critical need. Error tracking and answer review remain necessary even when the first records are relevant.

Sources, editions and credits

Original CIE explanations, six documents, five needs, judgments and scenarios. Conceptual references: Stanford CS276 and NIST TREC’s reciprocal-rank definition. No source chart, ranking example or code is reproduced. The CIE program and its unavailable-value conventions are not a complete trec_eval implementation.