TL;DR — Quick answer
A/B testing compares two versions of a page or element by splitting live traffic between them to see which converts better with real visitors. Change one thing at a time, decide your success metric and sample size in advance, and run the test until it reaches statistical significance before acting on the winner. On lower-traffic South African sites, test bigger, bolder changes and be patient with sample size, because small tweaks rarely produce readable results.
Key takeaways
- A/B testing replaces opinion with evidence from your real visitors
- Test one change at a time so you know what caused any difference
- Decide your metric and required sample size before you start
- Wait for statistical significance: stopping early gives false winners
- On low-traffic SA sites, test bold changes rather than tiny tweaks
- Keep a record of every test: even losers teach you about your audience
A/B testing, also called split testing, is a method of comparing two versions of a web page or element by showing each to a portion of your visitors and measuring which performs better. Instead of arguing about what will work, you let real behaviour decide. For South African businesses, especially those with modest traffic, knowing how to test correctly keeps the results trustworthy and the decisions sound.
What is A/B testing and how does it work?
A/B testing works by randomly splitting your visitors between two versions of a page, version A (the control) and version B (the variation), then measuring which produces more of your desired action. Because visitors are assigned at random, any difference in results can be attributed to the change you made.
For example, you might test two headlines, two button colours or two form lengths. Half your traffic sees each, and after enough visitors and conversions you compare the rates. The version that performs better, once the result is statistically reliable, becomes your new standard. It is the closest thing marketing has to a controlled experiment.
What should you test first?
Test the elements with the biggest potential impact on conversion first: headlines, your main call to action, page layout, form length and key offers. Testing trivial details before the big levers wastes traffic and time.
| Element | Why it matters |
|---|---|
| Headline | First thing visitors read; shapes the whole impression |
| Call to action | Wording and prominence directly affect clicks |
| Form length | Fewer fields usually lift completion |
| Offer or value prop | A clearer benefit can transform results |
| Page layout | Order and emphasis guide visitor attention |
A/B testing produces reliable results only when you change one element at a time, define your success metric and sample size in advance, and run the test to statistical significance before acting on the winner. Source: Juicy Designs, 2026.
How much traffic and time do you need?
You need enough visitors and conversions for the result to reach statistical significance, typically meaning a clear, reliable difference that is unlikely to be down to chance. The exact number depends on your baseline conversion rate and the size of the difference you want to detect.
As a rule of thumb, larger expected differences and higher conversion rates need less traffic, while small differences need much more. Decide your required sample size before you begin using a sample-size calculator, then commit to running the test for at least one full business cycle, usually a week or two, to avoid day-of-week distortions. Never stop the moment a version looks ahead, as early leads frequently reverse.
What A/B testing mistakes should you avoid?
The most common mistakes are testing several changes at once, stopping too early, ignoring statistical significance, and testing trivial elements. Each one leads you to act on results that are not real.
- Changing multiple things at once. You cannot tell which change caused the result.
- Stopping early. Calling a winner before significance gives false positives.
- Ignoring sample size. Small samples produce noisy, unreliable outcomes.
- Testing tiny details. On modest traffic, only big changes move the needle readably.
- Not segmenting. Mobile and desktop visitors can behave very differently.
How do you A/B test on low traffic?
On low-traffic South African sites, test bold, high-impact changes rather than small tweaks, be patient with the timeline, and lean on proven best practice where testing is not viable. Small tests simply cannot reach significance on limited traffic.
Focus your limited traffic on changes large enough to produce a clear difference, such as a completely different headline, offer or page structure. Accept that tests will run for weeks rather than days. Where you genuinely lack the traffic to test, apply established conversion principles instead, then measure the overall trend. At Juicy Designs we design and run pragmatic testing programmes for South African businesses, matched to the traffic they actually have.