Resources · 76
Published numbers: verify sources, calculations and interpretation
An editorial review procedure separating documented statistics, fictional examples, targets and commercial promises.
· 4 min
What this guide helps achieve
- Trace the original measurement
- Reproduce the calculation
- Classify the evidence
- Correct dependent content
Quick check
- Is a famous source sufficient?
- Does a small p-value establish a large improvement?
- Must all old numbers be removed?
Step-by-step method
- 01
Trace the original measurement
Follow reposts back to the document describing collection. Record organization, date, country, population, sample size and question. Multiple blogs repeating one number remain one source. Missing methodology should prevent a statistic from being treated as an established fact.
Deliverable: original reference and collection limits.
- 02
Reproduce the calculation
Write down numerator, denominator, exclusions, unit and period. Check rounding and totals. Fictional example: a rate rises from 2% to 3%. The difference is one percentage point; the relative increase is 50%. The same change produces different impressions when described in these two ways.
Deliverable: reproducible calculation.
- 03
Classify the evidence
An internal target is not an observed result. A teaching example is not a client case. Association alone does not establish causation. For experiments, inspect protocol, effect size, uncertainty and stopping rules before interpreting a result.
Deliverable: editorial status of the claim.
- 04
Correct dependent content
Remove or rephrase values that cannot be justified without inventing replacement percentages. Find translations, callouts, social images, exports and structured data repeating the claim. Keep a dated note when the correction changes what a reader would understand.
Deliverable: correction log and list of updated copies.
Acceptance case to reproduce
Fictional example: these inputs describe no customer or observed result.
View case inputs
{
"population": 200,
"eligible": 80,
"positive": 20,
"rate_all": 0.1,
"rate_eligible": 0.25
}Expected decision
20/200 is 10%, while 20/80 is 25%. Name the chosen denominator in the publication. The 80 eligible people are a subset of the 200 here: retain the exclusion reasons for the other 120. The 15-point difference between 10% and 25% comes from the chosen scope, without establishing a change in the phenomenon.
Verification procedure
- Record source, period, unit and eligible population.
- Recalculate the rate for each proposed denominator.
- Keep formula and exclusions beside the published number.
Understand · try · decide
Simpson's paradox: the total can reverse the comparison
A version can improve within every request profile yet show a lower overall rate. Inspect batch composition before concluding that performance regressed.
Fictional example: two different batches of 100 requests. Routine and complex profiles are defined before evaluation; success means a validated correct answer. This is not a randomized trial.
Every bar uses the same 0–100% scale. Counts appear in the table.
- Routine requests
A90 %B95 %
- Complex requests
A20 %B30 %
- Observed total
A83 %B43 %
Successes and counts by profile
| Profile | Version A: correct / tested | Version B: correct / tested |
|---|---|---|
| Routine requests | 81 / 90 · 90 % | 19 / 20 · 95 % |
| Complex requests | 2 / 10 · 20 % | 24 / 80 · 30 % |
| Observed total | 83 / 100 · 83 % | 43 / 100 · 43 % |
From observation to decision
- Keep the same success and profile definitions
- Show successes, counts and composition for each batch
- Choose a reference population before interpreting the difference
Calculation rule
w = p / 100 ; A(w) = 90w + 20(1 − w) ; B(w) = 95w + 30(1 − w)
Reusable worksheet
Complete with your authorised observations. These fields are a working template, not observed results.
| Field | Information to record |
|---|---|
| Claim | Exact sentence, page and copies |
| Measurement | Original source, population, date and method |
| Calculation | Numerator, denominator, unit and rounding |
| Decision | Publish, qualify or remove; owner and date |
Worked example
Illustrative situation
Fictional example: a page claims higher sales after a redesign without a study reference or comparison period.
Decision and expected evidence
The team removes the percentage, keeps the documented actions and publishes a measurement protocol for evaluating its own redesign.
Distinguish the mechanisms
| Mechanism | Purpose | Check or limitation |
|---|---|---|
| Observed measurement | Describe a population and period | State methodology and uncertainty |
| Fictional example | Explain a calculation or decision | Label its fictional status beside the values |
| Target | Define a desired outcome | Do not present it as achieved performance |
Management indicators
| Indicator | What it measures | First action |
|---|---|---|
| Traceability | Decisive numbers connected to original measurements | Review priority commercial claims |
| Correction coverage | Copies updated after a correction | Check languages, images and exports |
Common pitfalls
- Comparing only overall rates
- Averaging percentages without their counts
- Presenting standardization as causal evidence
Frequently asked questions
Is a famous source sufficient?
No. Verify that it actually supports the quoted number and that its population matches your sentence.
Does a small p-value establish a large improvement?
No. It measures neither effect size nor business importance. Examine test assumptions and uncertainty too.
Must all old numbers be removed?
No. Historical data can remain useful when its date and scope are visible and it is not presented as current.
