How do you A/B test a landing page?
What to test, how long to run it, and how to avoid calling a winner too early.
Test one meaningful change at a time, run in whole weeks, define the winning metric as qualified leads rather than clicks, and decide the stopping rule before you launch. Most A/B tests fail not because the idea was bad but because the test was called early.
What is worth testing
Big swings, not button colors. The changes that actually move conversion rate are the headline and its match to the ad, the offer itself, form length, the presence and placement of proof, and whether the page answers the visitor's main objection. A different shade of blue is a rounding error you will never be able to detect.
Test one variable per test. Changing the headline and the form at the same time means a result you cannot explain, which means you learned nothing you can reuse.
How long to run it
Two rules. Run for whole weeks, because Tuesday traffic and Saturday traffic behave differently and a test that starts Wednesday and ends Monday is weighted toward one kind of visitor. And run until you hit the sample size you calculated before starting.
That calculation is worth doing yourself rather than trusting a rule of thumb. Here is the shape of it: the smaller the effect you want to detect, the more conversions per variant you need, and the requirement grows roughly with the square of how small that effect is. Detecting a jump from 4 percent to 8 percent takes a modest sample. Detecting 4 percent to 4.4 percent takes a very large one. Use any free sample size calculator, enter your current rate and the smallest lift that would be worth acting on, and it will tell you the number of visitors per variant. If that number exceeds a couple of months of traffic, do not run the test. Make the change and monitor instead.
How to avoid calling it early
Peeking at a running test and stopping the moment it looks significant is the most common way to manufacture a fake winner. Early in a test the numbers swing wildly, and if you check daily you will eventually catch a swing. Set the end date and the sample size in advance, then look once.
Also define the metric before you start. If you decide afterward which number counts, you will pick the one that won.
HubSpot CMS has A/B testing built into pages and emails, so on a HubSpot site this does not require a separate tool or a script that slows the page down.
Related: How do you decide what to test first?, What is conversion rate optimization?, our blog post on A/B testing for small businesses, and our data and analytics services.