Because the split is random, a difference big enough to be unlikely by chance can be credited to the change itself. That causal claim is the whole value. Analytics can show two things moved together, but only an experiment supports saying one caused the other.
Think of it like this. A farmer wanting to know if a fertiliser works does not treat the whole field. They treat half, leave half alone, and compare, because otherwise a good harvest could just have been good weather.
How it works
Users are randomly assigned to version A or version B, and the assignment sticks so nobody sees both. One metric is chosen in advance. At the end, the metric is compared between groups.
The conventional bar is a p-value under 0.05, meaning a result this extreme would happen by chance less than one time in twenty if there were no real difference.
Sample size is decided before starting. This matters more than it sounds, and the reason is in the trade-offs below.
Trade-offs
- Significance is not size. With enough traffic, a difference too small to care about becomes statistically significant. Whether it is worth shipping is a separate judgement.
- It tells you what, never why. A variant can lift signups while quietly damaging trust, and the test will report a win because trust was not the metric.
- It needs traffic. Detecting a small improvement takes far more visitors than most products have. Low volume is exactly where convincing false positives come from.
Common mistakes
- Peeking. Checking repeatedly and stopping the moment the line crosses the threshold will manufacture significance out of noise. The maths assumes one check at a predetermined sample size.
- Reading a null result as proof. No significant difference may simply mean the test was too small to detect one.
- Testing without a hypothesis. Running variants to see what sticks produces winners that do not repeat.
Frequently asked questions
Key takeaways
- Fix the sample size before starting, then run to it.
- Significant and meaningful are different questions.
- A win on your chosen metric may be a loss on one you did not measure.
- Low traffic makes false positives more likely, not less.