Resources · 14
AI vendor evaluation: evidence, dependencies and exit conditions
A selection framework for testing real quality, qualifying data flows and preserving a practical exit option.
· 3 min
What this guide helps achieve
- Compare providers on representative cases
- Expose data flows and subprocessors
- Monitor service changes
- Test export and replacement before dependency
Quick check
- What business result is actually required?
- Which data leaves the system?
- Are limitations tested or merely stated?
- Who notifies model changes?
- Can the service be exported and replaced?
Step-by-step method
- 01
Frame needs and prohibited uses
Describe tasks, users, decision level, consequences of error and cases that remain outside the service. Do not select the solution before the need.
Deliverable: use, risk and owner brief.
- 02
Map data and dependencies
List inputs, outputs, logs, locations, retention, possible training, subprocessors, models and critical components.
Deliverable: data-flow and service-chain map.
- 03
Require verifiable evidence
Request relevant architecture, applicable policies, reports, incident process, continuity, change history and known limitations. Date every item.
Deliverable: evidence file and unconfirmed areas.
- 04
Test your cases
Build a versioned set covering common, sensitive, ambiguous, adversarial and unanswerable cases. Measure quality, stability, abstention, latency and cost.
Deliverable: comparable results and thresholds.
- 05
Control operation and change
Define access, monitoring, necessary logs, human validation, notifications, reassessment rights and regression handling.
Deliverable: production-control plan.
- 06
Exercise the exit
Test export of data and configuration, deletion, replacement, continuity and migration time before the service becomes indispensable.
Deliverable: executed exit scenario.
Worked example
Illustrative situation
Two AI offers are compared on the same anonymised set of real requests, including costly errors and cases with no possible answer. Claims about data, model changes and exit are matched to dated evidence.
Decision and expected evidence
Selection depends on results by case type, an owner for production oversight and an export that has actually been tried.
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Accepted cases | Representative cases above defined thresholds | Reject averages that conceal risk |
| Verified claims | Commitments linked to dated evidence | Qualify unsupported statements |
| Changes evaluated | Versions replayed on the same protocol | Block critical regressions |
| Reversibility | Assets exportable and restorable elsewhere | Test remaining dependencies |
Common pitfalls
- Comparing only a sales demonstration
- Accepting a certification name without scope
- Ignoring silent service changes
- Negotiating exit after deployment
Frequently asked questions
Is a vendor questionnaire enough?
No. It structures collection, but decisive points need documents, tests, clauses and context-appropriate checks.
Must the exact model be disclosed?
You mainly need to understand the capabilities, limits, changes and responsibilities that affect your use. Expected detail increases with risk.
When should reversibility be tested?
Before commitment and after major change. An untested exit clause remains an assumption.
Media: sanitization and evidence
Compare the supplier’s deletion promise with evidence
A data-deleted clause may cover an application operation, copies or media retirement. Request explicit scope and reconciliation between items delivered to a provider and those actually processed.
- Identify method, technique, tool and medium behind each attestation.
- Request verification results and the validation decision separately.
- For cloud storage, examine encrypted boundaries, key management and copies under contractual responsibilities.
Read the methods, evidence points and annotated situations →
NIST · SP 800-88 Rev. 2 · Guidelines for Media Sanitization · Final · 2025-09-26 · consulted 2026-10-23
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
Continue exploring
Related analysis
