Resources · 50

Small samples: communicate measurement uncertainty

Show counts, period and limitations before interpreting a rate or recommending action.

· 5 min

Empty cyber intelligence operations room at night AI-generated editorial illustration

What this guide helps achieve

  • Publish counts
  • Separate variation from collection faults
  • Name statistical assumptions
  • Avoid misleading precision

Quick check

  • What is the denominator?
  • Is one person counted repeatedly?
  • Are periods comparable?
  • Who is missing from the sample?
  • Which difference would change the decision?

Step-by-step method

  1. 01

    Define the unit

    State whether a rate concerns visits, people, orders or responses. Show successes and eligible units. A scope change can alter the rate without a behavioural change.

    Deliverable: definition and counts.

  2. 02

    Check collection and selection

    Reconcile results with source events. Identify consent effects, absent devices, voluntary responses, deduplication and partial periods. A statistical interval cannot repair a broken counter or excluded participants.

    Deliverable: coverage limitations.

  3. 03

    Choose an appropriate method

    For a binomial proportion, NIST describes Wilson and exact intervals among other methods. Check independence and sampling assumptions first. An interval calculated from repeated visits can imply unjustified precision.

    Deliverable: method, level and assumptions.

  4. 04

    Present estimate and range together

    Show counts, period and method beside the estimate. In frequentist interpretation, confidence describes the repeated procedure, not the probability that a fixed parameter lies in this particular interval.

    Deliverable: contextualised result.

  5. 05

    Avoid comparison shortcuts

    Do not infer a difference from interval overlap alone. Comparison requires an appropriate method, a plan and a predefined useful difference; repeated monitoring of results needs accounting for.

    Deliverable: comparison protocol.

  6. 06

    Decide with limitations

    Distinguish a cheap reversible action from a lasting commitment. Report inconclusive results and name useful additional evidence. Do not select a favourable conclusion merely because the test must end.

    Deliverable: decision and missing information.

Four situations for testing the method

Fictional teaching cases. Each situation connects a signal, a check and a decision bounded by evidence; none represents a client outcome.

Fictional situationConcrete checkDecision and limit
A KPI must show its relevant count — A dashboard ranks a team’s success rate from few requests alongside a team handling many more.Show numerator, denominator, period and success definition. Identify repeated observations and exclusions before comparison.Include uncertainty and collection limits. A ranking position does not establish a real difference in quality.
Repeated visits are not independent people — A small group returns frequently and creates many visits, but the report treats each visit as a new independent customer.Separate observation and decision units. Describe authorised identity rules and examine visit-frequency distributions per person or account where possible.Do not use visit count alone as an independent sample size. Retain limitations from unknown identities and multiple devices.
Do not choose a landing page from a handful of conversions — Two variants each obtain few conversions, and a dashboard declares a winner as soon as their rates diverge.Retain counts, allocation rule, period, useful difference and comparison plan defined before viewing results. Check collection and populations.Report unresolved uncertainty and follow the planned protocol. An observed gap or eyeballed intervals are insufficient to select a variant.
Show the baseline in a data story — A story announces a 100% increase when the result moves from one success in twenty to two in twenty.Show both counts, the 5% and 10% rates and the five-percentage-point gap. Describe period comparability and the analysis method.Separate observed change, uncertainty and interpretation. Relative doubling does not establish that the studied intervention caused it.

Understand · try · decide

A zero rate does not establish zero risk

Compare zero events in 20 observations with zero in 200. Both estimates are zero, but the uncertainty differs.

This calculation concerns a binomial proportion with a defined unit, independent observations and a stable probability in the population studied. Convenience sampling, repeated visits or incomplete collection can invalidate the interpretation.

The two-sided 95% Wilson interval expresses uncertainty under those assumptions. It does not predict the next outcome, correct selection bias or compare two variants here.

Reference points and evidence to retain

Reference pointWhat you can concludeEvidence to retain
0 / 20 0% observed; interval about 0 to 16.1% No observed events remains compatible with a nonzero population proportion.
0 / 200 0% observed; interval about 0 to 1.9% More observations narrow the interval under the same assumptions.
10 / 100 10% observed; interval about 5.5 to 17.4% Show the denominator before adding decimal places.

From observation to decision

  1. Define the unit
  2. Check independence and coverage
  3. Count events and observations
  4. Read rate and interval together
  5. Decide within the limits

Calculation rule

p = x / n; d = 1 + z²/n; centre = (p + z²/(2n))/d; margin = z × √(p(1 − p)/n + z²/(4n²))/d; z = 1.959963984540054. Bounds are centre ± margin.

Reusable worksheet

Complete with your authorised observations. These fields are a working template, not observed results.

FieldInformation to record
MeasureSuccesses / eligible units; definition
ContextPeriod, population, exclusions and repetitions
MethodEstimator, interval, level and assumptions
ConclusionObserved, inconclusive, action and needed evidence

Worked example

Illustrative situation

Calculated fictional example: 10 successes out of 100 units gives 10 %. A 95 % Wilson interval with z = 1.96 is about 5.5 % to 17.4 %, under binomial assumptions.

Decision and expected evidence

The commentary reports 10/100, period and assumptions. It promises neither the same future rate nor an improvement caused by a campaign.

Distinguish the mechanisms

MechanismPurposeCheck or limitation
Observed rateDescribes the sampleDoes not establish precision alone
IntervalQuantifies uncertainty under assumptionsDoes not fix selection bias
Comparative testExamines a difference under a planDoes not alone establish business value

Management indicators

IndicatorWhat it measuresFirst action
Eligible countUnits actually observedShow alongside the rate
Interval widthMethod-dependent uncertaintyAvoid overprecise conclusions
CoverageIncluded and absent populationsInspect blind spots

Common pitfalls

  • Showing many decimals for a few responses
  • Treating repeated visits as independent people
  • Equating nonsignificance with no effect
  • Hiding inconclusive results

Frequently asked questions

Does a zero rate prove zero risk?

No. No event in a limited sample does not establish impossibility in the population.

Does an interval correct bias?

No. It describes uncertainty under assumptions; coverage and collection quality still need review.

Can rates be compared visually?

Use counts and an appropriate comparison method. Interval overlap alone is not a test.

Official references

References consulted: . The method and worksheet propose checks to adapt to your context; they do not constitute certification.