Resources · 90

Synthetic data: verify privacy and task utility

Evaluate generated records before sharing them or using them to train a model.

· 5 min

Private AI computing infrastructure in an empty laboratory AI-generated editorial illustration

What this guide helps achieve

  • Specify what must be preserved
  • Map potential disclosure
  • Separate development from evaluation
  • Test disclosure scenarios
  • Choose a sharing model
  • Make the comparison reproducible
  • Reassess combined outputs

Quick check

  • Does synthetic mean anonymous?
  • Should rare cases be preserved?
  • Is one score enough?
  • Does low similarity to source rows prove anonymity?
  • Can each export be approved independently?

Step-by-step method

  1. 01

    Specify what must be preserved

    Choose a task: testing a form, comparing an algorithm or reproducing distributions. Record useful variables, relationships and rare cases. Tables that look similar do not establish task utility.

    Output: Useful-variable and case specification.

  2. 02

    Map potential disclosure

    Record source data, generator access, direct identifiers and rare combinations. Replacing names often amounts to pseudonymisation. A synthetic label alone does not establish anonymisation.

    Output: Source and risk map.

  3. 03

    Separate development from evaluation

    Keep an authorised real reference set outside generator tuning. Compare the task and subpopulations using a predefined method. Never copy real records into public examples.

    Output: Independent versioned protocol.

  4. 04

    Test disclosure scenarios

    In an authorised environment, investigate near copies, rare values and linkage with other datasets. Record attacker assumptions and access. A negative test is not a universal guarantee.

    Output: Scenario and limitation report.

  5. 05

    Choose a sharing model

    Compare public release, restricted access and protected environments. Approve recipients, limits and retention. Prepare withdrawal and reassessment when available sources or linkage techniques change.

    Output: Sharing and reassessment decision.

  6. 06

    Make the comparison reproducible

    Fix the task, authorized reference, subgroups and success rule before testing. Compare real and synthetic data with the same control inputs, units and windows. Report counts, failures and missing data per subgroup; unmeasured results remain unknown. Set thresholds for the intended use instead of adopting one universal aggregate score.

    Comparison table with subgroup counts and limits.

  7. 07

    Reassess combined outputs

    Inventory released versions, recipients and authorized query exports. Check whether combining information makes rare cases easier to link or infer. A new version must not inherit the previous approval automatically. Document access, withdrawal and limitations: removing a download does not erase copies already delivered.

    Release register and new decision on combinations.

Fictional acceptance-test scenarios

These proposed cases are not client observations. Adapt data, permissions and acceptance criteria to your authorised environment.

Situation to exerciseResult to checkEvidence to retain
A rare combination appears in a synthetic dataset.Examine possible linkage to accessible data; generated origin alone does not establish anonymity.Linkage scenario, access used, outcomes and limitations.
Means are preserved but a small subgroup disappears.Compare authorised use-case outcomes by subgroup and disclose lost utility.Group definitions, counts and relevant differences.
A new dataset is generated from a changed source.Repeat utility and privacy checks before release; an old assessment does not establish current safety.Source version, test procedure and release decision.

Acceptance case to reproduce

Fictional example: these inputs describe no customer or observed result.

View case inputs
{
    "synthetic": true,
    "rare_source_tuple": {
        "age": 97,
        "postcode": "00000",
        "role": "unique"
    },
    "same_tuple_in_output": true
}

Expected decision

The rare tuple survives. Test linkage risks before claiming the dataset is anonymous.

Fictional variations to test

The synthetic dataset has the right average, but small groups disappear.

Also evaluate the groups and tasks actually intended. Good aggregate utility does not guarantee reliable results for every subpopulation.

The public version is synthetic, but diagnostic files contain source data.

Inspect the entire released package: logs, examples, metadata and supplementary files. The main table’s status does not automatically cover those elements.

Two versions acceptable separately reveal information when combined.

Assess joint releases and accessible auxiliary data. Review risk before the next release; evaluating the first version is not enough.

Evidence to retain

  • Utility results by task and group, with documented limitations.
  • Inventory of distributed files and checks of their contents.
  • Compared versions, linkage assumptions and release decision.

Additional reference : NIST — SP 800-188, De-Identifying Government Datasets [EN]

Verification procedure

  1. Record sources, rare attributes and authorized uses.
  2. Check close combinations against authorized source data.
  3. Keep linkage risks and task utility evidence.
  4. Run the same fictional task on an authorized reference and synthetic data; retain subgroup outcomes and counts.
  5. Combine two authorized fictional exports to examine their joint information without publishing real records.
  6. Change a source variable or recipient and verify a new documented decision precedes release.

CNIL — Anonymisation

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
UseTask, recipients and necessary variables
EvaluationIndependent reference, segments and criteria
DisclosureAuthorised scenarios, outcomes and limits
DecisionAccess, owner and review date

Synthetic data: compare subgroups and released versions

An aggregate score can hide poor utility in a small group. Fictional example: 90 successes in 100 cases, but only 2 in a rare subgroup’s 10 cases. These are not customer results. They show why decisions need denominators and separate utility from disclosure risk.

Worked example

Illustrative situation

Fictional example: generated applicant records reproduce a rare combination of location and career history.

Decision and expected evidence

The team removes that case from release, tests remaining utility and selects restricted access instead of open publication.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
MaskingRemove obvious identifiersCombinations may remain identifying
Synthetic generationCreate artificial recordsAssess memorisation and utility
Protected accessRestrict users and outputsTest exports too

Management indicators

IndicatorWhat it measuresFirst action
Near copiesSimilarity to source recordsReview affected cases
Segment utilityPerformance on the defined taskReport underrepresented groups
Permitted outputsExports actually allowedTest restrictions
Subgroup difference from referenceOutcome difference for a defined task and subgroupShow counts, direction and unmeasured results
Released versions reassessedVersions with combined-risk decision / reviewed versionsState recipients and access assumptions

Common pitfalls

  • Combinations may remain identifying
  • Assess memorisation and utility
  • Test exports too

Frequently asked questions

Does synthetic mean anonymous?

No. Assess context, techniques and reidentification possibilities; a marketing label is not evidence.

Should rare cases be preserved?

Only according to the task and risk. Document the trade-off and evaluate rare populations separately.

Is one score enough?

No. Utility, privacy and coverage answer different questions. Keep their results separate.

Does low similarity to source rows prove anonymity?

No. Distance choice, available attributes and linkage opportunities affect the result. Retain assumptions and scenarios; one score does not replace contextual evaluation.

Can each export be approved independently?

Also examine what several exports or versions reveal together. Release scope and information available to recipients are part of the decision.

Official references

References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.