A team runs a test. On day three the new variant is up 18% and the tool says 95% confident. They ship it, write it up, and the conversion rate does not move.
Nobody did anything dishonest. The test was simply never capable of answering the question, and the number on the dashboard gave no hint of that. Most of what goes wrong in conversion testing is arithmetic that happened before the test started.
Significance does not mean what the dashboard implies
Statistical significance is the probability of seeing a difference at least this large if there were genuinely no difference at all. The convention is a p-value of 0.05 or below, meaning roughly a one-in-twenty chance of a false alarm.
Three things it does not mean, all of which get assumed daily:
- It is not the probability that your variant is better. That is a different quantity requiring different assumptions.
- It says nothing about size. A significant result can be a 0.2% lift that is worth nothing commercially.
- It is only valid if the test was run as designed. Stopping early or peeking repeatedly breaks the guarantee entirely.
Sample size, decided before you start
This is the step that gets skipped, and skipping it is what makes the rest meaningless. Required sample depends on your baseline conversion rate and the smallest lift worth detecting.
At a 3% baseline, 80% power and 95% significance, here is roughly what each ambition costs you:
| Relative lift you want to detect | Visitors needed per variant |
|---|---|
| 10% (3.0% → 3.3%) | ~53,000 |
| 20% (3.0% → 3.6%) | ~13,900 |
| 50% (3.0% → 4.5%) | ~2,500 |
Read that table twice, because it reframes the whole activity. Detecting a small improvement is enormously more expensive than detecting a large one — halving the effect you want to catch roughly quadruples the traffic you need.
If you get 4,000 visitors a month, you cannot detect a 10% lift. Not slowly — at all, in any reasonable timeframe. That is not a reason to give up on testing; it is a reason to test bolder changes. Small sites should be testing whole-page rewrites, not button colours.
Peeking is the one that gets everyone
You planned for 20,000 visitors per variant. On day two you are curious, so you look. Then again on day four. And day six.
Every look is another opportunity to catch a random fluctuation crossing the threshold. The 5% false positive rate assumes you evaluate the result once, at the end. Check repeatedly and stop the moment it goes significant, and the real false positive rate climbs well past 5% — with enough peeks, a test between two identical pages will eventually show a significant winner.
Stopping a test the moment it looks good is how you generate a portfolio of wins that never show up in revenue.
Two legitimate ways out. Either fix the sample size in advance and genuinely do not act until you reach it, or use a testing tool built on sequential methods — always-valid p-values or a Bayesian approach — that is explicitly designed to permit continuous monitoring. What you cannot do is use a fixed-horizon calculation and then treat it as if it were sequential.
Run whole weeks
Traffic is not the same on a Tuesday morning as on a Saturday night, and neither is intent. A test that runs Monday to Thursday has measured your weekday audience and quietly generalised to everyone.
- Run in complete seven-day blocks, ending on the same weekday you started
- Two weeks minimum, even when the numbers arrive sooner
- Avoid running across a sale, a holiday or a campaign spike
- Discount the first few days for novelty — returning visitors react to change itself before they react to the change
Check the test was set up correctly
Before trusting any result, verify the mechanics. Broken instrumentation produces very confident nonsense.
- 1Sample ratio. A 50/50 split should land close to 50/50. A persistent skew — 52/48 across tens of thousands of sessions — means the assignment or tracking is broken, and the result is void regardless of what it says.
- 2One variable at a time, unless you are running a properly designed multivariate test and have the traffic for it.
- 3Assignment persists. A returning visitor must see the same variant. If not, you are measuring noise.
- 4Both variants actually work. Test the variant on mobile and in a second browser. A layout that breaks on iOS will lose, and you will conclude the wrong thing.
Testing many things at once
Run twenty tests at a 5% false positive rate and you should expect roughly one false winner purely by chance. Comparing five variants in a single test has the same problem, since each comparison gets its own roll of the dice.
Keep the number of variants small, decide your primary metric before you start rather than picking the one that looks best afterwards, and treat a surprising win as a hypothesis to re-test rather than a conclusion to ship.
What to actually test
Given how expensive small effects are to detect, aim big. In rough order of how often they move the number:
| Change | Typical effect |
|---|---|
| Removing form fields | Often large |
| Rewriting the value proposition above the fold | Often large |
| Changing page structure or the order of sections | Moderate to large |
| Pricing presentation and framing | Moderate to large |
| Adding proof — specific, credible, near the ask | Moderate |
| Button colour and microcopy | Usually undetectable |
Button colour tests survive because they are easy to set up, not because they work. At most realistic traffic levels the effect is far smaller than the test could ever detect, so the result is noise either way.
One genuinely underrated candidate: page speed. It is one of the few changes that can be tested against a real user population with an unambiguous mechanism behind it — what actually makes LCP slow covers the technical side.
Writing down the result
Record every test, including the flat ones. A repository of "we tried this and nothing happened" is more valuable after two years than a folder of wins, because it stops your team re-running the same idea every eighteen months.
- The hypothesis, written before the test ran
- The planned sample size and duration
- The primary metric, chosen in advance
- The result, with the confidence interval — not just the point estimate
- The decision, and whether it was later confirmed in the actual numbers
That last line is the honest one, and the one almost nobody keeps. If your winning tests do not eventually show up in the monthly figures, something in the process is producing wins that are not real.
Conversion Rate Optimization
Data-driven conversion rate optimization — testing, analytics, and funnel improvements that turn more of your existing visitors into customers. Free consultation.
See how we run it →