Short answer
A test requires a specific hypothesis, a sufficiently large user sample, uniform random assignment, and changing one variable at a time, then confirming statistical significance to ensure the result is not due to chance.
Explanation
A p-value is the probability of observing a conversion-rate difference this large or larger between design A and B if the change had no real effect (the null hypothesis). If it is below the preset significance level (usually 0.05), reject the null hypothesis. A common mistake is interpreting this as "a 95% chance that design B is better"; a p-value measures neither the probability that the hypothesis is true nor the effect size. Stopping the test after two days just because one version is ahead is the notorious Peeking Problem. Run the test through complete weekly cycles to capture behavior differences between weekdays and weekends.
Common mistakes
- Changing 10 different elements in version B at once, making it impossible to tell which element caused a result change.
- Stopping the test before reaching the statistically calculated sample size, creating false positives.