Resources · 49
Evaluate AI regressions before changing versions
Compare versions on the same tasks, find local losses and make a decision supported by cases rather than an average.
· 6 min
What this guide helps achieve
- Isolate the tested change
- Preserve an independent reference set
- Inspect failures by segment
- Prepare shutdown or rollback
Quick check
- Which component changed?
- Was the test set used to optimise the candidate?
- Who resolves annotation disagreements?
- Does an average hide a sensitive error?
- Can the previous configuration be restored?
Step-by-step method
- 01
Define task and refusals
Describe expected output, permitted data and costly errors. Separate accuracy, usefulness, appropriate refusal and action effects. NIST’s framework calls for context-appropriate measurement; it does not provide a universal reliability score.
Deliverable: criteria and decision owner.
- 02
Separate reference and discovery
Keep a stable set out of optimisation. Add new anonymised feedback cases separately: they reveal blind spots but should not silently change the comparison denominator.
Deliverable: versioned sets with provenance.
- 03
Document annotation
Explain what makes an answer acceptable, partial or incorrect. Have a competent reviewer resolve ambiguity and retain reasons for disagreements. A single reference answer may be too restrictive for an open-ended task.
Deliverable: rubric and adjudicated cases.
- 04
Compare under equal conditions
Replay identical inputs with recorded versions, parameters, tools and sources. Repeat trials when outputs vary and report variation. Separate model failure, absent data, unavailable tools and a defective test.
Deliverable: paired outcomes and minimised traces.
- 05
Decide by risk family
Inspect losses by language, document type and sensitive situation even when the mean improves. Define stop, correction and limited-release criteria before reading results. One severe case can justify withholding a release.
Deliverable: decision, reservations and exceptions.
- 06
Replay after release
Prepare a restorable configuration and reachable owner. Compare early feedback with the test set without treating local success as evidence for all uses. Reassess when data or dependencies change.
Deliverable: monitoring and rollback procedure.
Four situations for testing the method
Fictional teaching cases. Each situation connects a signal, a check and a decision bounded by evidence; none represents a client outcome.
| Fictional situation | Concrete check | Decision and limit |
|---|---|---|
| Compare models on consequential errors — A classifier improves its overall score but becomes worse at identifying requests needing urgent handling. | Replay identical cases, retain results by request type and review critical errors using a common definition. | Decide using severity, volume and recovery capability. The best average score is not automatically the best service choice. |
| Check what transfer learning preserves and degrades — A model adapted to a new domain succeeds on recent examples but regresses on historical categories still in use. | Keep a historical reference set separate from training and add new-domain cases. Compare versions on both sets. | Document trade-offs and exclusions before release. Faster training does not establish preservation of required capabilities. |
| Do not average away a disparity — A change reduces errors for a majority group while increasing errors for a minority group; the aggregate looks better. | Report counts, group definitions and error consequences. Replay affected cases using reviewed labels and comparable conditions. | Include local losses in the decision. An aggregate result is insufficient to establish fairness or identify the cause of a gap. |
| Avoid evaluating a model only on its generator’s logic — A model is trained on synthetic examples and evaluated only on examples produced by the same process. | Use an independent authorised control set, compare rare cases and examine which constraints the generator reproduces or omits. | Report coverage limits. Success on synthetic data establishes neither real-case performance nor privacy of the generated records. |
Understand · try · decide
Compare the same cases: net gains and local regressions
Two overall rates do not identify which cases changed. In this fictional example, each task is judged acceptable or unacceptable for A and B using the same criterion set before testing.
An appropriate refusal may be acceptable for the task. Cases missing a judgment for either version stay separate; they are not treated as failures.
| A / B | B acceptable | B unacceptable |
|---|---|---|
| A acceptable | 70Both acceptable | 10Losses from A to B |
| A unacceptable | 15Gains from A to B | 5Both unacceptable |
Calculation rule
N = c + l + g + e ; A% = 100(c + l)/N ; B% = 100(c + g)/N ; Δpp = 100(g − l)/N
Interpret transitions
What the rate cannot decide
These fictional variations are independent of the values entered in the calculator.
- Pair
Keep case identifiers, versions and test conditions aligned.
- Judge
Use the same criterion and resolve missing or ambiguous judgments.
- Inspect
Review losses by risk family before deciding.
Fictional reading exercises, without evaluating a real system.
Same rate: same acceptable answers?
- Observation
- Another batch has ten gains and ten losses.
- Expected interpretation
- The net difference is zero, but twenty cases change status.
A judgment for B is missing?
- Observation
- A is judged; B is not yet judged.
- Expected interpretation
- Keep the pair outside the complete table, report the omission and obtain the judgment.
The annotation rule changed?
- Observation
- A uses the old criterion and B the new one.
- Expected interpretation
- Rejudge both versions using a common rule before comparing.
Just one critical loss?
- Observation
- One of the ten losses violates an isolation rule defined as blocking before testing.
- Expected interpretation
- Hold release for that scope; fifteen gains do not remove the blocker.
Reusable worksheet
Complete with your authorised observations. These fields are a working template, not observed results.
| Field | Information to record |
|---|---|
| Case and segment | Stable identifier, language, task type |
| Expected | Acceptance criterion and reference evidence |
| Versions A / B | Outcome, repetition, failure cause |
| Decision | Accept, correct or stop; owner and evidence |
Worked example
Illustrative situation
Fictional example: a release improves routine answers but drops uncertainty statements when documents conflict.
Decision and expected evidence
The report isolates that subset, retains source evidence and withholds release for that use until a correction is retested.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| Stable set | Compare versions | Keep out of optimisation |
| New cases | Find blind spots | Report separately from historical results |
| In-service observation | Understand actual use | Respect permissions, minimisation and context |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Paired losses | Previously accepted cases now failing | Inspect each sensitive loss |
| Annotation disagreement | Cases with differing judgements | Clarify the rubric before comparison |
| Segment coverage | Situations actually represented | Name untested uses |
Common pitfalls
- Optimising on the final test
- Changing set and version together
- Treating an automated judge as truth
- Accepting critical losses because the mean improves
Frequently asked questions
Is an automated judge required?
No. It can speed up triage if its rubric is validated; ambiguous or costly cases still need business adjudication.
Is a mean improvement enough?
No. Compare relevant cases and segments, then apply the criteria established before testing.
How many cases are needed?
Volume depends on diversity and risk. Report count, provenance and limitations; no small set establishes general safety.
RAG: access rights, citations and caches
Replay rights after every RAG-chain change
Changing a lexical engine, fusion, cache or citation resolver may reopen a secondary path. Freeze a fictional document set with expected outcomes before comparing versions.
- Version policy, index, connector, model and cases.
- Exercise initial, revoked-group and deleted-object states.
- Examine every difference without hiding a critical failure in an average.
Explore the RAG access matrix and replay cases →
References consulted on · OWASP · LLM08:2025 Vector and Embedding Weaknesses OWASP · API1:2023 Broken Object Level Authorization
AI: check a JSON result
Test refused and undecidable outputs too
A model, prompt or validator update can change handling of duplicates, types and dates. Acceptance checks need to cover these decisions beyond the number of parseable JSON responses.
- Version the schema, validator settings and reference cases.
- Check that well-formed but false values remain blocked.
- Keep syntax, contract, date and evidence errors separate.
Compare the 14 outputs and their reasons →
Sources reviewed on · IETF · RFC 8259 · JSON Schema · Object reference · JSON Schema · Draft 2020-12 validation · Python 3.12 · json decoder · python-jsonschema · Schema validation · OWASP · LLM05:2025 Improper Output Handling
RAG relevance and ranking
Compare retrieval before evaluating the answer
Keep the same needs, judgments and depth k when comparing configurations. Inspect regressing queries before drawing conclusions from an average.
- Separate window precision, relevant-record coverage and ordering quality.
- Document missing judgments and the no-relevant-record convention.
- Then evaluate the answer, citations and refusals using a separate protocol.
Explore ranking measures and relevance judgments →
Stanford CS276 · Retrieval evaluation · 2019 course archive · PDF pp. 3, 5–6 · consulted 2026-10-23
NIST TREC · Reciprocal rank definition · Repository snapshot 35cf096ba6d5ae08ca4803c056b31ed61a56a6d4 · consulted 2026-10-23
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
- NIST — AI Risk Management Framework 1.0, Measure
- NIST — AI RMF Core, Measure
- Dror et al. · ACL 2018 · §3.2.2, paired observations
- OWASP · LLM08:2025 Vector and Embedding Weaknesses
- OWASP · API1:2023 Broken Object Level Authorization
- IETF · RFC 8259
- JSON Schema · Object reference
- JSON Schema · Draft 2020-12 validation
- Python 3.12 · json decoder
- python-jsonschema · Schema validation
- OWASP · LLM05:2025 Improper Output Handling
- Stanford CS276 · Retrieval evaluation
- NIST TREC · Reciprocal rank definition
