What Is Split Testing?
Split testing is a controlled experiment in which incoming traffic is divided, typically 50/50, between a control version (the original) and a variation (the challenger). Both versions run simultaneously under the same conditions and traffic sources.
After a statistically significant number of visitors have been exposed to each version, the results are compared to determine which performs better against the predefined success metric, most commonly conversion rate.
The strength of split testing lies in its isolation of variables. When you change only one element between the control and the variation, any difference in performance can be confidently attributed to that change.
Common elements to test include the headline, the call-to-action button text and colour, the hero image, the form length, the pricing presentation, and the placement of trust signals such as testimonials or security badges.
Each test builds institutional knowledge about what your specific audience responds to.
Split testing differs from multivariate testing in scope and traffic requirements. A split test asks a single question: "Does version A or version B convert better-" A multivariate test asks multiple questions simultaneously, exploring how different combinations of elements interact.
For most South African businesses that do not have the volume of traffic needed for multivariate significance, split testing is the more practical and reliable approach.
Tools such as Google Optimise (now succeeded by A/B testing functionality in GA4 and third-party platforms), VWO, and Optimizely all support straightforward split testing without requiring developer involvement for every test.
Statistical significance is a critical concept in split testing. A result is considered statistically significant when you can be confident, typically at the 95% level, that the observed difference in conversion rates is not due to random chance.
Running a test for too short a period, or with too little traffic, produces misleading results that lead to bad decisions.
South African businesses with lower monthly traffic volumes should plan tests to run for at least four weeks to capture weekly behavioural patterns and collect enough data for reliable conclusions.
Split Testing In Practice
The two scenarios below are illustrative examples, not Juicy Designs client results. The figures indicate the scale of effect that split testing typically produces, so treat them as indicative rather than measured.
Picture a Johannesburg-based insurance comparison website that notices its quote request form has a low completion rate. Rather than redesigning the entire page on instinct, the team runs a split test. Version A is the current page with a five-field form visible immediately.
Version B moves the form below a brief value proposition section and reduces it to three fields. After three weeks and around 4,000 visitors per variant, Version B could plausibly show a completion rate around 20% higher, at a statistical significance level above the usual 95% threshold.
A test of this kind would typically pay for itself within the first month.
Or consider a Cape Town e-commerce store selling beauty products, where a split test might reveal that a CTA button reading "Shop Now" outperforms "Add to Cart" on category pages by something in the region of 15%. An insight like that is counterintuitive, which is exactly why it has to come from a test rather than from instinct.
Visitors on category pages are still browsing, and "Shop Now" feels less committal than "Add to Cart", which implies a purchase decision has already been made.
This kind of locally validated insight, gathered from your own South African audience, is far more valuable than best practices borrowed from overseas case studies that may not reflect local consumer behaviour and trust patterns.
Running a test that produces a trustworthy answer
- Start from a hypothesis, not a hunch: what you will change, what you expect to happen, and why.
- Change one variable. Bundled changes tell you the result but not the cause.
- Split traffic simultaneously, not sequentially, so day-of-week and campaign changes affect both variants equally.
- Decide the sample size before starting, and do not stop the moment a variant looks ahead.
- Run full weeks so weekday and weekend behaviour are both represented.
- Record losers as well as winners. Knowing what does not work prevents repeating it.
Split testing on low traffic
The uncomfortable arithmetic: detecting a small improvement reliably needs a large sample. Most South African SME sites cannot reach that in a reasonable period, which is why so many published tests are called early and wrong.
The honest alternatives at low volume:
- Test bigger changes. A substantially different page produces effects large enough to detect.
- Test higher up the funnel, where volume is greater: ad creative and headlines rather than button colour.
- Use qualitative evidence. Session recordings and five customer interviews often identify the blocker faster than a test would confirm it.
- Fix obvious problems without testing. A twelve-field form does not need an experiment to justify shortening.
See our A/B testing guide and multivariate testing.
FAQ
How much traffic do I need to run a valid split test?
A reliable split test typically requires at least 1,000 visitors per variant to achieve statistical significance at the 95% confidence level. South African websites with lower traffic volumes should run tests for longer periods, ideally four to eight weeks, rather than ending them early based on small sample sizes.
What is the difference between split testing and multivariate testing?
Split testing compares two complete versions of a page, changing one major element at a time. Multivariate testing tests multiple elements simultaneously to see how different combinations interact. Split testing requires less traffic and is simpler to implement, making it the right starting point for most South African businesses.
How long should a split test run?
Long enough to reach the sample size you calculated before starting, and always in complete weeks so weekday and weekend behaviour are both represented. Stopping when a variant first looks ahead is the most common cause of false results.
What if I do not have enough traffic to split test?
Test bigger changes, test higher up the funnel where volume is greater, and lean on qualitative research such as session recordings and customer interviews. Fix clearly broken things without testing them at all.