Resources · 76

Published numbers: verify sources, calculations and interpretation

An editorial review procedure separating documented statistics, fictional examples, targets and commercial promises.

· 4 min

Empty intelligence archive with dossiers and maps AI-generated editorial illustration

What this guide helps achieve

  • Trace the original measurement
  • Reproduce the calculation
  • Classify the evidence
  • Correct dependent content

Quick check

  • Is a famous source sufficient?
  • Does a small p-value establish a large improvement?
  • Must all old numbers be removed?

Step-by-step method

  1. 01

    Trace the original measurement

    Follow reposts back to the document describing collection. Record organization, date, country, population, sample size and question. Multiple blogs repeating one number remain one source. Missing methodology should prevent a statistic from being treated as an established fact.

    Deliverable: original reference and collection limits.

  2. 02

    Reproduce the calculation

    Write down numerator, denominator, exclusions, unit and period. Check rounding and totals. Fictional example: a rate rises from 2% to 3%. The difference is one percentage point; the relative increase is 50%. The same change produces different impressions when described in these two ways.

    Deliverable: reproducible calculation.

  3. 03

    Classify the evidence

    An internal target is not an observed result. A teaching example is not a client case. Association alone does not establish causation. For experiments, inspect protocol, effect size, uncertainty and stopping rules before interpreting a result.

    Deliverable: editorial status of the claim.

  4. 04

    Correct dependent content

    Remove or rephrase values that cannot be justified without inventing replacement percentages. Find translations, callouts, social images, exports and structured data repeating the claim. Keep a dated note when the correction changes what a reader would understand.

    Deliverable: correction log and list of updated copies.

Acceptance case to reproduce

Fictional example: these inputs describe no customer or observed result.

View case inputs
{
    "population": 200,
    "eligible": 80,
    "positive": 20,
    "rate_all": 0.1,
    "rate_eligible": 0.25
}

Expected decision

20/200 is 10%, while 20/80 is 25%. Name the chosen denominator in the publication. The 80 eligible people are a subset of the 200 here: retain the exclusion reasons for the other 120. The 15-point difference between 10% and 25% comes from the chosen scope, without establishing a change in the phenomenon.

Verification procedure

  1. Record source, period, unit and eligible population.
  2. Recalculate the rate for each proposed denominator.
  3. Keep formula and exclusions beside the published number.

NIST — Confidence intervals

Understand · try · decide

Simpson's paradox: the total can reverse the comparison

A version can improve within every request profile yet show a lower overall rate. Inspect batch composition before concluding that performance regressed.

Fictional example: two different batches of 100 requests. Routine and complex profiles are defined before evaluation; success means a validated correct answer. This is not a randomized trial.

Compare profiles and the total

Every bar uses the same 0–100% scale. Counts appear in the table.

  • Routine requests
    A90 %
    B95 %
  • Complex requests
    A20 %
    B30 %
  • Observed total
    A83 %
    B43 %

Successes and counts by profile

ProfileVersion A: correct / testedVersion B: correct / tested
Routine requests 81 / 90 · 90 % 19 / 20 · 95 %
Complex requests 2 / 10 · 20 % 24 / 80 · 30 %
Observed total 83 / 100 · 83 % 43 / 100 · 43 %

From observation to decision

  1. Keep the same success and profile definitions
  2. Show successes, counts and composition for each batch
  3. Choose a reference population before interpreting the difference

Calculation rule

w = p / 100 ; A(w) = 90w + 20(1 − w) ; B(w) = 95w + 30(1 − w)

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
ClaimExact sentence, page and copies
MeasurementOriginal source, population, date and method
CalculationNumerator, denominator, unit and rounding
DecisionPublish, qualify or remove; owner and date

Worked example

Illustrative situation

Fictional example: a page claims higher sales after a redesign without a study reference or comparison period.

Decision and expected evidence

The team removes the percentage, keeps the documented actions and publishes a measurement protocol for evaluating its own redesign.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
Observed measurementDescribe a population and periodState methodology and uncertainty
Fictional exampleExplain a calculation or decisionLabel its fictional status beside the values
TargetDefine a desired outcomeDo not present it as achieved performance

Management indicators

IndicatorWhat it measuresFirst action
TraceabilityDecisive numbers connected to original measurementsReview priority commercial claims
Correction coverageCopies updated after a correctionCheck languages, images and exports

Common pitfalls

  • Comparing only overall rates
  • Averaging percentages without their counts
  • Presenting standardization as causal evidence

Frequently asked questions

Is a famous source sufficient?

No. Verify that it actually supports the quoted number and that its population matches your sentence.

Does a small p-value establish a large improvement?

No. It measures neither effect size nor business importance. Examine test assumptions and uncertainty too.

Must all old numbers be removed?

No. Historical data can remain useful when its date and scope are visible and it is not presented as current.

Official references