A controlled experiment comparing two versions to see which performs better.
An A/B test (split test) randomly divides your audience between a control version (A) and one variant (B), then measures which produces a better result on a chosen metric. Randomisation is what makes it powerful: it isolates the effect of the change from everything else going on, giving you causal evidence rather than correlation.
The test is only trustworthy if it is set up properly: a single clear hypothesis, a predefined primary metric, a sample large enough to detect a meaningful difference, and a run long enough to reach statistical significance before you decide.
Traffic is split randomly and simultaneously so both versions face the same conditions. You then compare the primary metric and check whether the difference is statistically significant or just noise.
Peeking at results and stopping the moment a variant looks ahead, testing during unrepresentative periods (a sale, a holiday), changing several elements so you cannot tell what caused the effect, or ignoring the false-positive risk of running many tests at once.
Common questions
Straight answers on how this fits your marketing and build.
Long enough to reach a predetermined sample size and statistical significance, and ideally covering full weekly cycles to avoid day-of-week bias. Stopping early because a variant looks ahead is the most common way tests mislead.
Still have questions? Talk to a specialist
Get a free audit and a 6-month roadmap showing exactly where you stand vs your top 3 competitors — no fluff, no upsell.
Trusted by 500+ brands · 4.9/5 client rating