A/B Testing With Low Traffic: What to Do When You Cannot Reach Significance

If your site gets a few hundred visitors a week, most A/B tests will never tell you anything. That is not a reason to stop improving the site. It is a reason to stop pretending a split test is the right tool, and to use ones that work at your scale.

A/B testing with low traffic fails for a simple reason: the maths needs volume. Evan Miller's sample size calculator is the standard way to check. Its own worked example takes a 10.2% baseline conversion rate and asks for a lift to 13.2%, and the answer is 2,545 visitors per variation. That is a large jump on a high baseline. Most service business sites convert at a fraction of that rate and are hoping for smaller gains, which pushes the required numbers far higher.

Why underpowered tests mislead you

You will call winners that are noise. Stop a test early because variant B is ahead after nine days and you are mostly reading chance. The result feels like evidence and is not.

You will miss real improvements. A genuine 15% lift on a page with 40 enquiries a month is invisible. The test ends "inconclusive" and the idea gets binned.

You will burn months. A test that needs six months to reach a sample size has delayed every other change on that page by six months.

Run the calculator before you build anything. Put in your real baseline and the smallest lift you would act on. If the answer is more visitors than you will see in two months, do not run the test.

What to do instead

Fix the obvious things without testing them. A form with eleven fields, a page that takes six seconds to load on mobile, a phone number that is not clickable, a headline that does not say what you do. None of these need a split test. You already know the answer. Our post on improving enquiry form conversion covers the form fixes, and page speed covers the rest.

Test bigger changes, not smaller ones. Small changes need huge samples to detect. Button colours and headline tweaks produce effects too small to measure at your volume. Rebuild the offer, restructure the page, change the form from five steps to one. Large differences show up in small samples, or at least show up as clear before-and-after movement.

Change one page at a time and compare periods carefully. A before-and-after comparison is weaker than a split test, but it is honest if you hold other things steady: same traffic source, same budget, no promotions running, and at least four full weeks on each side. Write down what changed and when. Treat the result as a signal, not a proof.

Watch behaviour, not just conversions. Session recordings and heatmaps show where people stall, rage-click or leave. Watch twenty recordings of the page that matters and you will find three problems you can fix this week. This is qualitative research, and at low volume it beats statistics.

Talk to the people who did not convert. Ask the enquiries who went quiet why. Ask the ones who did convert what nearly stopped them. Five conversations will give you a better test hypothesis than any tool.

Move the test to where the volume is. If your paid campaigns send ten times the traffic of your organic pages, test there first. Landing page changes on paid traffic reach a sample faster because you control the volume. Our Google Ads management work often starts by testing the landing page against the ad, not the other way round. Google Ads also has its own experiments feature for testing campaign settings such as bidding, which is a separate job from testing page design.

If you do run a test, protect your rankings

Split testing can cause SEO problems if it is set up badly. Google's guidance on minimising A/B testing impact in Search is short and worth reading. The key points are:

• Do not show Googlebot a different version from human visitors. Google treats that as cloaking, a spam policy violation.
• Put rel="canonical" on any alternate test URLs, pointing at the original.
• Use 302 redirects, not 301s, when sending users to a variant.
• Run the test only as long as it needs and remove the test code when it ends. Google may treat a test that runs unnecessarily long as an attempt to deceive search engines.

That last point matters for low-traffic sites in particular. The temptation is to leave a test running for a year hoping it eventually reaches significance. If it needs that long, it was the wrong test.

A sensible order of work

Start with fixes that need no proof: speed, forms, clarity, contact options. Move to one large change at a time with a clean before-and-after read. Use recordings and customer conversations to decide what to change next. Save formal split testing for the point where a page reliably sees thousands of visitors per variation, because that is when the statistics start to work in your favour.

If you would like a second opinion on which changes are worth making on your site first, our conversion rate optimisation service starts with exactly that. Get in touch and we will tell you plainly whether testing is the right next step or whether you should simply fix the page.