Resources · 49

Evaluate AI regressions before changing versions

Compare versions on the same tasks, find local losses and make a decision supported by cases rather than an average.

· 6 min

Empty cyber intelligence operations room at night AI-generated editorial illustration

What this guide helps achieve

  • Isolate the tested change
  • Preserve an independent reference set
  • Inspect failures by segment
  • Prepare shutdown or rollback

Quick check

  • Which component changed?
  • Was the test set used to optimise the candidate?
  • Who resolves annotation disagreements?
  • Does an average hide a sensitive error?
  • Can the previous configuration be restored?

Step-by-step method

  1. 01

    Define task and refusals

    Describe expected output, permitted data and costly errors. Separate accuracy, usefulness, appropriate refusal and action effects. NIST’s framework calls for context-appropriate measurement; it does not provide a universal reliability score.

    Deliverable: criteria and decision owner.

  2. 02

    Separate reference and discovery

    Keep a stable set out of optimisation. Add new anonymised feedback cases separately: they reveal blind spots but should not silently change the comparison denominator.

    Deliverable: versioned sets with provenance.

  3. 03

    Document annotation

    Explain what makes an answer acceptable, partial or incorrect. Have a competent reviewer resolve ambiguity and retain reasons for disagreements. A single reference answer may be too restrictive for an open-ended task.

    Deliverable: rubric and adjudicated cases.

  4. 04

    Compare under equal conditions

    Replay identical inputs with recorded versions, parameters, tools and sources. Repeat trials when outputs vary and report variation. Separate model failure, absent data, unavailable tools and a defective test.

    Deliverable: paired outcomes and minimised traces.

  5. 05

    Decide by risk family

    Inspect losses by language, document type and sensitive situation even when the mean improves. Define stop, correction and limited-release criteria before reading results. One severe case can justify withholding a release.

    Deliverable: decision, reservations and exceptions.

  6. 06

    Replay after release

    Prepare a restorable configuration and reachable owner. Compare early feedback with the test set without treating local success as evidence for all uses. Reassess when data or dependencies change.

    Deliverable: monitoring and rollback procedure.

Four situations for testing the method

Fictional teaching cases. Each situation connects a signal, a check and a decision bounded by evidence; none represents a client outcome.

Fictional situationConcrete checkDecision and limit
Compare models on consequential errors — A classifier improves its overall score but becomes worse at identifying requests needing urgent handling.Replay identical cases, retain results by request type and review critical errors using a common definition.Decide using severity, volume and recovery capability. The best average score is not automatically the best service choice.
Check what transfer learning preserves and degrades — A model adapted to a new domain succeeds on recent examples but regresses on historical categories still in use.Keep a historical reference set separate from training and add new-domain cases. Compare versions on both sets.Document trade-offs and exclusions before release. Faster training does not establish preservation of required capabilities.
Do not average away a disparity — A change reduces errors for a majority group while increasing errors for a minority group; the aggregate looks better.Report counts, group definitions and error consequences. Replay affected cases using reviewed labels and comparable conditions.Include local losses in the decision. An aggregate result is insufficient to establish fairness or identify the cause of a gap.
Avoid evaluating a model only on its generator’s logic — A model is trained on synthetic examples and evaluated only on examples produced by the same process.Use an independent authorised control set, compare rare cases and examine which constraints the generator reproduces or omits.Report coverage limits. Success on synthetic data establishes neither real-case performance nor privacy of the generated records.

Understand · try · decide

Compare the same cases: net gains and local regressions

Two overall rates do not identify which cases changed. In this fictional example, each task is judged acceptable or unacceptable for A and B using the same criterion set before testing.

An appropriate refusal may be acceptable for the task. Cases missing a judgment for either version stay separate; they are not treated as failures.

Fictional reference case: 100 jointly judged tasks
A / BB acceptableB unacceptable
A acceptable 70Both acceptable 10Losses from A to B
A unacceptable 15Gains from A to B 5Both unacceptable

Calculation rule

N = c + l + g + e ; A% = 100(c + l)/N ; B% = 100(c + g)/N ; Δpp = 100(g − l)/N

Interpret transitions

What the rate cannot decide

These fictional variations are independent of the values entered in the calculator.

  1. Pair

    Keep case identifiers, versions and test conditions aligned.

  2. Judge

    Use the same criterion and resolve missing or ambiguous judgments.

  3. Inspect

    Review losses by risk family before deciding.

Fictional reading exercises, without evaluating a real system.

Same rate: same acceptable answers?
Observation
Another batch has ten gains and ten losses.
Expected interpretation
The net difference is zero, but twenty cases change status.
A judgment for B is missing?
Observation
A is judged; B is not yet judged.
Expected interpretation
Keep the pair outside the complete table, report the omission and obtain the judgment.
The annotation rule changed?
Observation
A uses the old criterion and B the new one.
Expected interpretation
Rejudge both versions using a common rule before comparing.
Just one critical loss?
Observation
One of the ten losses violates an isolation rule defined as blocking before testing.
Expected interpretation
Hold release for that scope; fifteen gains do not remove the blocker.

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
Case and segmentStable identifier, language, task type
ExpectedAcceptance criterion and reference evidence
Versions A / BOutcome, repetition, failure cause
DecisionAccept, correct or stop; owner and evidence

Worked example

Illustrative situation

Fictional example: a release improves routine answers but drops uncertainty statements when documents conflict.

Decision and expected evidence

The report isolates that subset, retains source evidence and withholds release for that use until a correction is retested.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
Stable setCompare versionsKeep out of optimisation
New casesFind blind spotsReport separately from historical results
In-service observationUnderstand actual useRespect permissions, minimisation and context

Management indicators

IndicatorWhat it measuresFirst action
Paired lossesPreviously accepted cases now failingInspect each sensitive loss
Annotation disagreementCases with differing judgementsClarify the rubric before comparison
Segment coverageSituations actually representedName untested uses

Common pitfalls

  • Optimising on the final test
  • Changing set and version together
  • Treating an automated judge as truth
  • Accepting critical losses because the mean improves

Frequently asked questions

Is an automated judge required?

No. It can speed up triage if its rubric is validated; ambiguous or costly cases still need business adjudication.

Is a mean improvement enough?

No. Compare relevant cases and segments, then apply the criteria established before testing.

How many cases are needed?

Volume depends on diversity and risk. Report count, provenance and limitations; no small set establishes general safety.

RAG: access rights, citations and caches

Replay rights after every RAG-chain change

Changing a lexical engine, fusion, cache or citation resolver may reopen a secondary path. Freeze a fictional document set with expected outcomes before comparing versions.

  • Version policy, index, connector, model and cases.
  • Exercise initial, revoked-group and deleted-object states.
  • Examine every difference without hiding a critical failure in an average.

Explore the RAG access matrix and replay cases →

References consulted on · OWASP · LLM08:2025 Vector and Embedding Weaknesses OWASP · API1:2023 Broken Object Level Authorization

AI: check a JSON result

Test refused and undecidable outputs too

A model, prompt or validator update can change handling of duplicates, types and dates. Acceptance checks need to cover these decisions beyond the number of parseable JSON responses.

  • Version the schema, validator settings and reference cases.
  • Check that well-formed but false values remain blocked.
  • Keep syntax, contract, date and evidence errors separate.

Compare the 14 outputs and their reasons →

Sources reviewed on · IETF · RFC 8259 · JSON Schema · Object reference · JSON Schema · Draft 2020-12 validation · Python 3.12 · json decoder · python-jsonschema · Schema validation · OWASP · LLM05:2025 Improper Output Handling

RAG relevance and ranking

Compare retrieval before evaluating the answer

Keep the same needs, judgments and depth k when comparing configurations. Inspect regressing queries before drawing conclusions from an average.

  • Separate window precision, relevant-record coverage and ordering quality.
  • Document missing judgments and the no-relevant-record convention.
  • Then evaluate the answer, citations and refusals using a separate protocol.

Explore ranking measures and relevance judgments →

Stanford CS276 · Retrieval evaluation · 2019 course archive · PDF pp. 3, 5–6 · consulted 2026-10-23

NIST TREC · Reciprocal rank definition · Repository snapshot 35cf096ba6d5ae08ca4803c056b31ed61a56a6d4 · consulted 2026-10-23

Official references

References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.