A/B Testing

What it is

A/B testing is a controlled experiment in which eligible users, requests, or other experimental units are randomly assigned to a control variant and one or more treatment variants. Teams compare a predefined primary metric while monitoring guardrail metrics for harmful side effects.

Core design decisions

• State a falsifiable hypothesis before looking at results.
• Choose the experimental unit and assignment method.
• Define a primary outcome, guardrails, and a minimum detectable effect.
• Keep assignment stable and avoid contamination between variants.
• Run long enough to cover normal traffic cycles and reach adequate statistical power.
• Analyze practical significance as well as statistical significance.

Feature flags commonly deliver variants, but a flag alone is not an experiment. A valid test also needs randomization, exposure logging, metric definitions, and an analysis plan.

Common failure modes

Peeking repeatedly without correction, changing metrics after results appear, sample-ratio mismatch, novelty effects, overlapping experiments, and measuring only short-term conversion can all produce misleading conclusions.