A/B testing: a beginner's guide
A/B testing is a method of comparing two versions of a web page or element by splitting traffic between them, so you can see which one converts better with real visitors rather than guessing. Test one change at a time, run the test until you reach statistical significance, and only then act on the winner.
Opinions about what will convert better are cheap and usually wrong. A/B testing replaces guesswork with evidence by letting your real visitors decide which version of a page performs best. For South African businesses with limited traffic, knowing how to test properly, and when not to, is what keeps the results trustworthy.
TL;DR: Quick Answer
A/B testing compares two versions of a page or element by splitting live traffic between them to see which converts better with real visitors. Change one thing at a time, decide your success metric and sample size in advance, and run the test until it reaches statistical significance before acting on the winner. On lower-traffic South African sites, test bigger, bolder changes and be patient with sample size, because small tweaks rarely produce readable results.
Key takeaways
- A/B testing replaces opinion with evidence from your real visitors
- Test one change at a time so you know what caused any difference
- Decide your metric and required sample size before you start
- Wait for statistical significance: stopping early gives false winners
- On low-traffic SA sites, test bold changes rather than tiny tweaks
- Keep a record of every test: even losers teach you about your audience
A/B testing, also called split testing, is a method of comparing two versions of a web page or element by showing each to a portion of your visitors and measuring which performs better. Instead of arguing about what will work, you let real behaviour decide. For South African businesses, especially those with modest traffic, knowing how to test correctly keeps the results trustworthy and the decisions sound.
What is A/B testing and how does it work?
A/B testing works by randomly splitting your visitors between two versions of a page, version A (the control) and version B (the variation), then measuring which produces more of your desired action. Because visitors are assigned at random, any difference in results can be attributed to the change you made.
For example, you might test two headlines, two button colours or two form lengths. Half your traffic sees each, and after enough visitors and conversions you compare the rates. The version that performs better, once the result is statistically reliable, becomes your new standard. It is the closest thing marketing has to a controlled experiment.
What should you test first?
Test the elements with the biggest potential impact on conversion first: headlines, your main call to action, page layout, form length and key offers. Testing trivial details before the big levers wastes traffic and time.
| Element | Why it matters |
|---|---|
| Headline | First thing visitors read; shapes the whole impression |
| Call to action | Wording and prominence directly affect clicks |
| Form length | Fewer fields usually lift completion |
| Offer or value prop | A clearer benefit can transform results |
| Page layout | Order and emphasis guide visitor attention |
A/B testing produces reliable results only when you change one element at a time, define your success metric and sample size in advance, and run the test to statistical significance before acting on the winner. Source: Juicy Designs, 2026.
How much traffic and time do you need?
You need enough visitors and conversions for the result to reach statistical significance, typically meaning a clear, reliable difference that is unlikely to be down to chance. The exact number depends on your baseline conversion rate and the size of the difference you want to detect.
As a rule of thumb, larger expected differences and higher conversion rates need less traffic, while small differences need much more. Decide your required sample size before you begin using a sample-size calculator, then commit to running the test for at least one full business cycle, usually a week or two, to avoid day-of-week distortions. Never stop the moment a version looks ahead, as early leads frequently reverse.
What A/B testing mistakes should you avoid?
The most common mistakes are testing several changes at once, stopping too early, ignoring statistical significance, and testing trivial elements. Each one leads you to act on results that are not real.
- Changing multiple things at once. You cannot tell which change caused the result.
- Stopping early. Calling a winner before significance gives false positives.
- Ignoring sample size. Small samples produce noisy, unreliable outcomes.
- Testing tiny details. On modest traffic, only big changes move the needle readably.
- Not segmenting. Mobile and desktop visitors can behave very differently.
How do you A/B test on low traffic?
On low-traffic South African sites, test bold, high-impact changes rather than small tweaks, be patient with the timeline, and lean on proven best practice where testing is not viable. Small tests simply cannot reach significance on limited traffic.
Focus your limited traffic on changes large enough to produce a clear difference, such as a completely different headline, offer or page structure. Accept that tests will run for weeks rather than days. Where you genuinely lack the traffic to test, apply established conversion principles instead, then measure the overall trend. At Juicy Designs we design and run pragmatic testing programmes for South African businesses, matched to the traffic they actually have.
Frequently asked questions
What is A/B testing in simple terms?
A/B testing compares two versions of a web page or element by splitting live traffic between them and measuring which converts better. Version A is the control and version B is the variation. Because visitors are assigned at random, any difference in results can be attributed to the change you made.
What should you A/B test first on your website?
Start with the elements that have the biggest potential impact on conversion: your headline, main call to action, page layout, form length and key offer. Testing trivial details before these big levers wastes traffic and time, especially on South African sites with modest visitor numbers.
Why should you only test one change at a time?
Testing one change at a time means you know exactly what caused any difference in results. If you change the headline, button and layout together, you cannot tell which element did the work. Isolating a single variable keeps your test a fair, controlled experiment.
How long should an A/B test run for?
Run the test until it reaches statistical significance and for at least one full business cycle, usually a week or two, to avoid day-of-week distortions. Decide your required sample size in advance with a calculator, and never stop the moment one version pulls ahead, because early leads frequently reverse.
Can you run A/B tests on a low-traffic website?
Yes, but focus on bold, high-impact changes such as a completely different headline, offer or page structure, because small tweaks rarely reach statistical significance on limited traffic. Accept that tests will run for weeks rather than days, and apply proven conversion principles where testing is genuinely not viable.