资源 · 26

运行支持合理决策的 A/B 测试。

在比较不同版本之前,明确假设、分配、主要指标和限制条件。

已更新 · 12 min

虚构的无人测量实验室:两台相同的黑色设备、黄铜灯和酒红色笔记本 由人工智能生成的编辑插图

本指南有助于实现的目标

  • 预注册协议
  • 检查分配情况
  • 关注安全措施
  • 在不确定性下做出决策

快速检查

  • 哪些具体假设可能被证伪?
  • 哪个单元接收变体?
  • 观察到的分配是否与计划相符?
  • 哪些损害需要暂停?
  • 持续时间和细分是否在看到结果之前设定?

分步指南

  1. 1

    陈述拟议的变更

    写出预期机制、目标人群以及对一项主要指标的影响。 描述可能的可访问性和服务风险。

    交付成果:带有日期的假设和成功标准。

  2. 2

    定义分配

    选择随机化单元,并在访问期间保持分配的稳定性。 在发布前排除机器人并指定超出范围的细分。

    交付成果:分配协议和排除项。

  3. 3

    仪器和测试

    验证两种变体的暴露、转化和防护措施事件。 测试移动端、拒绝同意和错误状态。

    交付成果:两种流程的质量保证记录。

  4. 4

    检查样本比例

    在读取处理效果之前,比较观察到的分配和预期分配。 不匹配可能暴露分配、测量或目标定位方面的缺陷。

    交付成果:比例检查和差异诊断。

  5. 5

    按计划时间进行分析

    遵守选定的时间窗口,并报告不确定性、主要指标、危害和计划的细分市场分析。 检查意外情况,不要事后只选择有利的数字。

    交付成果:包含局限性和不确定性的分析。

  6. 6

    决策和跟进

    为推广、放弃或其他测试指定负责人。 推广后,检查效果和防护措施是否保持一致。

    交付成果:已存档的决策和上线后检查。

管理指标

指标衡量内容首要行动
分配预期样本与观察样本比例分析前进行诊断
暴露情况实际暴露的合格用户修复目标
影响主要指标的变化和不确定性应用计划的决策规则
防护措施每个变体的错误、延迟和放弃情况暂停有害处理

常见错误

  • 在数值首次看起来有利时停止
  • 在没有主要指标的情况下收集多个指标
  • 忽略样本比例不匹配
  • 将小片段视为一般证明

常见问题解答

样本比例是否只是一个次要的形式?

否。无法解释的不匹配会使比较结果不可靠。

是否应该每日检查转化率?

运营监控有所帮助,但根据结果更改持续时间或决策规则会破坏原有的解读。

如果结果不确定怎么办?

报告不确定性和可能的​​效应量,然后决定是否需要进行其他实验。

A/B分配:数据与SRM

此语言版本的详细参考说明以英语提供。

A conversion difference becomes difficult to interpret when compared groups no longer match the assignment plan. Sample Ratio Mismatch (SRM) is a data quality signal: investigate allocation before evaluating the product change. This dossier connects calculation, instrumentation and decisions without treating a p-value as overall validation of an experiment.

In 2020 Microsoft Research describes four stages where SRM can arise and an ExP reference threshold p < 0.0005 using a chi-squared test. The educational CIE reader below uses a two-sided exact binomial test for two 50/50 groups. It does not reproduce ExP’s algorithm; the historical threshold is illustrative, not a universal standard.

Microsoft Research · Diagnosing Sample Ratio Mismatch · Published 2020-09-14 · consulted 2026-10-23

Four places to investigate a counting discrepancy

Four places to investigate a counting discrepancy
Stage documented by ExPCIE review questionOriginal practical investigation
AssignmentCheck the unit and rule that assign each participant.Replay the same identity across visits; compare configuration versions and change times. A reassigned unit no longer represents two independent observations.
ExecutionCompare what each variant actually allows to execute.Reproduce redirects, incomplete loading, errors and cancellation. Record when exposure is logged and whether the variant changes that moment.
Log processingReconcile inputs, exclusions and joins.Count units before and after each transformation. Check missing keys, duplicates, bot filters and telemetry loss by group.
AnalysisExamine the population retained for comparison.Compare the assigned population with the analysed subset. A filter triggered by an action available only in B selects a population affected by treatment.

Define what the two counts represent

Define what the two counts represent
Reader assumptionConsequence for interpretation
Two mutually exclusive variantsCount each unit once, in A or B. Repeated page views or events are not distinct participants.
Fixed 50/50 allocationEvery unit has the same probability of assignment to A and B. A 90/10 allocation, changing ratio or enforced quota needs a different model.
Independent trialsUnits linked through an organisation, household or reassignment can violate the binomial model. Choose the unit in the protocol.
Fixed window and filtersCompare the same interval and deduplication and eligibility rules. Do not adapt exclusions to obtain a desired result.
Bounded aggregate countsAccept 0 to 5,000 units per arm, at most 10,000 overall. This is an interface limit, not a minimum statistical power requirement.
Quality check onlyThe reader computes neither effect, conversion, power nor required sample size. A result without an SRM signal does not establish causality.

NIST/SEMATECH · Binomial distribution · e-Handbook §1.3.6.6.18 · consulted 2026-10-23

50/50分配检查

Enter the two observed counts or load a fictional example. This checks allocation, not conversion counts. Calculation stays in the browser, without transmission, storage, import or export.

Under H₀, X follows Binomial(N, 0.5), with N = A + B and k = min(A, B). The reader computes p = min(1, 2 × Σᵢ₌₀ᵏ C(N,i) / 2ᴺ). This doubles the smaller tail, valid here by 50/50 symmetry. Sums are evaluated in logarithms to retain very small probabilities; displayed values are rounded.

A p-value describes how this allocation compares with the assumed model; it is not the probability that the experiment is valid or that a bug exists. A small value does not locate the cause. A large value does not establish error-free data. Repeated checks and multiple segments change false-alert control: define a suitable monitoring rule before operational use.

Eight educational examples readable without JavaScript

R01 · Two balanced groups
Two balanced groups
Count ACount BExpected reading
5050p = 1 under the model. This checks neither instrumentation, conversion nor causality.
R02 · 60/40 across ten units
60/40 across ten units
Count ACount BExpected reading
64p = 0.75390625. A visibly different ratio alone does not establish a signal at this small sample size.
R03 · Same ratio, hundredfold sample
Same ratio, hundredfold sample
Count ACount BExpected reading
600400The ratio remains 60/40, but the p-value becomes very small. Sample size changes statistical interpretation.
R04 · One empty arm, ten units
One empty arm, ten units
Count ACount BExpected reading
010p = 0.001953125 remains above the illustrative 0.0005 threshold. No alert at this threshold does not certify both arms are working.
R05 · One empty arm, twenty units
One empty arm, twenty units
Count ACount BExpected reading
020p = 0.0000019073486328125: a signal under the reference threshold. Investigate before interpreting an effect.
R06 · No observed units
No observed units
Count ACount BExpected reading
00No p-value: no observations. Show unavailable data rather than a reassuring result.
R07 · Extremely small probability
Extremely small probability
Count ACount BExpected reading
50001Displaying log₁₀(p) avoids presenting numerical rounding to zero as a mathematical impossibility.
R08 · Leading space is not repaired
Leading space is not repaired
Count ACount BExpected reading
5050Input rejected. The reader accepts only ASCII integers from 0 to 5,000, without spaces, signs, separators or redundant leading zeros.

Before resuming effect analysis

Before resuming effect analysis
Suggested checkRecord or result to retain
TimelineDiscrepancy onset, assignment versions, deployments and collection changes.
Stage countsAssigned, exposed and analysed counts; differences explained using the same unit.
IdentitiesEvidence of unit stability and explicit treatment of duplicates, session changes and sign-ins.
TelemetryComparable events, delivery errors and losses by variant, device and version.
FilteringEligibility and exclusion rules, with their position before or after treatment.
DecisionSupported cause, fix, still-usable period or new experiment, owner and review date.

Avoid artificially balancing groups by deleting observations to remove an alert. Reconstruct the data path first. An effect metric, guardrails and uncertainty remain necessary after repairing counts. In the final brief, separate the statistical signal, demonstrated cause and product decision.

Sources, editions and credits

Sources: Microsoft Research, article dated 14 September 2020; NIST/SEMATECH, binomial distribution §1.3.6.6.18. Questions, recipes and eight cases are CIE educational material, not results from real experiments. No institutional chart, logo or visual is reproduced.

Microsoft Research · Diagnosing Sample Ratio Mismatch · Published 2020-09-14 · consulted 2026-10-23

SHA-256 : d28a6a66b3f1d014f3aafce6c0239b6a6477049b2d4582f06f4d6aaae651ee75

NIST/SEMATECH · Binomial distribution · e-Handbook §1.3.6.6.18 · consulted 2026-10-23

SHA-256 : d899609cde11a15ebd745f6313931da10e9511a0dff64f39b027e5b9ad103ad8

官方参考文献

参考文献支持该方法。 请根据您的实际情况调整检查;它们并非认证。 原始参考文献标题和源文档可能使用其他语言。

参考资料查阅日期 .