Resources · 90
Synthetic data: verify privacy and task utility
Evaluate generated records before sharing them or using them to train a model.
· 5 min
What this guide helps achieve
- Specify what must be preserved
- Map potential disclosure
- Separate development from evaluation
- Test disclosure scenarios
- Choose a sharing model
- Make the comparison reproducible
- Reassess combined outputs
Quick check
- Does synthetic mean anonymous?
- Should rare cases be preserved?
- Is one score enough?
- Does low similarity to source rows prove anonymity?
- Can each export be approved independently?
Step-by-step method
- 01
Specify what must be preserved
Choose a task: testing a form, comparing an algorithm or reproducing distributions. Record useful variables, relationships and rare cases. Tables that look similar do not establish task utility.
Output: Useful-variable and case specification.
- 02
Map potential disclosure
Record source data, generator access, direct identifiers and rare combinations. Replacing names often amounts to pseudonymisation. A synthetic label alone does not establish anonymisation.
Output: Source and risk map.
- 03
Separate development from evaluation
Keep an authorised real reference set outside generator tuning. Compare the task and subpopulations using a predefined method. Never copy real records into public examples.
Output: Independent versioned protocol.
- 04
Test disclosure scenarios
In an authorised environment, investigate near copies, rare values and linkage with other datasets. Record attacker assumptions and access. A negative test is not a universal guarantee.
Output: Scenario and limitation report.
- 05
Choose a sharing model
Compare public release, restricted access and protected environments. Approve recipients, limits and retention. Prepare withdrawal and reassessment when available sources or linkage techniques change.
Output: Sharing and reassessment decision.
- 06
Make the comparison reproducible
Fix the task, authorized reference, subgroups and success rule before testing. Compare real and synthetic data with the same control inputs, units and windows. Report counts, failures and missing data per subgroup; unmeasured results remain unknown. Set thresholds for the intended use instead of adopting one universal aggregate score.
Comparison table with subgroup counts and limits.
- 07
Reassess combined outputs
Inventory released versions, recipients and authorized query exports. Check whether combining information makes rare cases easier to link or infer. A new version must not inherit the previous approval automatically. Document access, withdrawal and limitations: removing a download does not erase copies already delivered.
Release register and new decision on combinations.
Fictional acceptance-test scenarios
These proposed cases are not client observations. Adapt data, permissions and acceptance criteria to your authorised environment.
| Situation to exercise | Result to check | Evidence to retain |
|---|---|---|
| A rare combination appears in a synthetic dataset. | Examine possible linkage to accessible data; generated origin alone does not establish anonymity. | Linkage scenario, access used, outcomes and limitations. |
| Means are preserved but a small subgroup disappears. | Compare authorised use-case outcomes by subgroup and disclose lost utility. | Group definitions, counts and relevant differences. |
| A new dataset is generated from a changed source. | Repeat utility and privacy checks before release; an old assessment does not establish current safety. | Source version, test procedure and release decision. |
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"synthetic": true,
"rare_source_tuple": {
"age": 97,
"postcode": "00000",
"role": "unique"
},
"same_tuple_in_output": true
}Expected decision
The rare tuple survives. Test linkage risks before claiming the dataset is anonymous.
Fictional variations to test
The synthetic dataset has the right average, but small groups disappear.
Also evaluate the groups and tasks actually intended. Good aggregate utility does not guarantee reliable results for every subpopulation.
The public version is synthetic, but diagnostic files contain source data.
Inspect the entire released package: logs, examples, metadata and supplementary files. The main table’s status does not automatically cover those elements.
Two versions acceptable separately reveal information when combined.
Assess joint releases and accessible auxiliary data. Review risk before the next release; evaluating the first version is not enough.
Evidence to retain
- Utility results by task and group, with documented limitations.
- Inventory of distributed files and checks of their contents.
- Compared versions, linkage assumptions and release decision.
Additional reference : NIST — SP 800-188, De-Identifying Government Datasets [EN]
Verification procedure
- Record sources, rare attributes and authorized uses.
- Check close combinations against authorized source data.
- Keep linkage risks and task utility evidence.
- Run the same fictional task on an authorized reference and synthetic data; retain subgroup outcomes and counts.
- Combine two authorized fictional exports to examine their joint information without publishing real records.
- Change a source variable or recipient and verify a new documented decision precedes release.
Reusable worksheet
Complete with your authorised observations. These fields are a working template, not observed results.
| Field | Information to record |
|---|---|
| Use | Task, recipients and necessary variables |
| Evaluation | Independent reference, segments and criteria |
| Disclosure | Authorised scenarios, outcomes and limits |
| Decision | Access, owner and review date |
Synthetic data: compare subgroups and released versions
An aggregate score can hide poor utility in a small group. Fictional example: 90 successes in 100 cases, but only 2 in a rare subgroup’s 10 cases. These are not customer results. They show why decisions need denominators and separate utility from disclosure risk.
Worked example
Illustrative situation
Fictional example: generated applicant records reproduce a rare combination of location and career history.
Decision and expected evidence
The team removes that case from release, tests remaining utility and selects restricted access instead of open publication.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| Masking | Remove obvious identifiers | Combinations may remain identifying |
| Synthetic generation | Create artificial records | Assess memorisation and utility |
| Protected access | Restrict users and outputs | Test exports too |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Near copies | Similarity to source records | Review affected cases |
| Segment utility | Performance on the defined task | Report underrepresented groups |
| Permitted outputs | Exports actually allowed | Test restrictions |
| Subgroup difference from reference | Outcome difference for a defined task and subgroup | Show counts, direction and unmeasured results |
| Released versions reassessed | Versions with combined-risk decision / reviewed versions | State recipients and access assumptions |
Common pitfalls
- Combinations may remain identifying
- Assess memorisation and utility
- Test exports too
Frequently asked questions
Does synthetic mean anonymous?
No. Assess context, techniques and reidentification possibilities; a marketing label is not evidence.
Should rare cases be preserved?
Only according to the task and risk. Document the trade-off and evaluate rare populations separately.
Is one score enough?
No. Utility, privacy and coverage answer different questions. Keep their results separate.
Does low similarity to source rows prove anonymity?
No. Distance choice, available attributes and linkage opportunities affect the result. Retain assumptions and scenarios; one score does not replace contextual evaluation.
Can each export be approved independently?
Also examine what several exports or versions reveal together. Release scope and information available to recipients are part of the decision.
Official references
References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.
