What is A/B testing?

When creating a marketing idea, for example, a landing page, it is tempting to work from your gut feeling. But what works for one person is not helpful for another. That is why it is wise to create 2 versions of your idea and compare them with each other. How do you do this? With an A/B test. In this article, you will read all about A/B testing and how to create 1 yourself in just 8 steps.

A/B testing meaning

A/B testing is a method where you randomly present 2 versions, version A and version B, to the target audience to see which version performs best. This could be, for example, a landing page, newsletter or design. With an A/B test, you find out which version positively impacts your business goals so that you can make a good choice.

These terms are also known as an A/B test:

• Split test
• Bucket test
• Split-run test

Good to know: A multivariate test is a test consisting of many A/B tests together.

How does A/B testing work?

In an A/B test, you create two different versions, from an entire page to a Call To Action button (CTA). Version A (control) is randomly shown to half of the website visitors, and version B (variation) to the other half of the visitors.

Over time, you have collected data from both versions. You compare this data to see which version performs best. For example, you collected the number of clicks, purchases or registrations. You implement the version that best suits your objectives.

An A/B testing example

We’ll give you an example to explain better how an A/B test works. Suppose you sell dog accessories and have a page about food bowls. This page has a blue CTA button with the ‘Buy now’ text. The conversions on this page are not running optimally, and you want to know whether the CTA button affects this. In this case, you can set up an A/B test to test, for example, two different colours with each other. Or the text ‘Buy now’ and ‘Order’. It is also possible to put the button in a different place and compare both places.

Why A/B testing?

A/B testing helps you discover whether your tactics are working or whether the target audience prefers a different version. But more reasons exist why you should start A/B testing:

1. Better user experience

The website user experience affects your goals. If visitors cannot find what they want, they will drop out. With an A/B test, you determine what works better and make your site more user-friendly.

2. More traffic

By testing different page titles, you will determine which title people are likelier to click on. This can help you get more traffic to your website.

3. More conversions

A CTA button has a lot of influence on whether or not it converts. Therefore, test different colours, spots and texts with each other, and find out which button converts best.

4. Fewer abandoned shopping carts

An abandoned shopping cart can mean the purchase process is not working optimally. Therefore, test different product photos, descriptions, checkout pages or even the location of shipping costs.

5. Lower bounce rate

With an A/B test, you can learn why people leave your website quickly (bounce). The text may be too difficult, the design doesn’t attract, or the colours don’t suit your target audience.

6. Useful dates

With an A/B test, you collect data, allowing you to substantiate why something works or not. So you don’t have to make decisions from guesswork or your gut feeling that may not be effective.

What can you A/B test?

You can test a lot during A/B testing. In the introduction, we already mentioned you can compare 2 landing pages, for example. There is more, for example:

• The subject line of an e-mail
• The title of a blog article
• A CTA button
• Colours and fonts
• Product photos
• Texts
• The navigation, such as a navigation menu
• Contact or registration forms
• The checkout process of your webshop
• Facebook or Instagram ads

How many visitors do you need for an A/B test?

How many visitors you need for an A/B test cannot be said precisely. Factors such as the desired effect and the conversion rate of the original version affect the number of visitors you need. In general, though, the more visitors, the more reliable the results will be.

Tip: With Optimizely’s A/B test sample size calculator, you can calculate how many visitors you need to run a reliable A/B test.

A/B testing tools

Several tools exist you can use to set up an A/B test. For example:

• VWO
• Optimizely
• Crazy Egg
• Kameleoon
• Omniconvert

Tip: Do you use WordPress? With a WordPress plugin, you can run an A/B test, for example, with Nelio AB Testing and Split Hero.

How to do A/B testing?

To find out what works well and what doesn’t, you’ll naturally want to start an A/B test. Below is a roadmap that will help you create your A/B test in 8 steps.

1. Collect data

First, you need to know how your website is doing now. With an analytics tool like Google Analytics, you can get insight into which pages you need to optimise. For example, choose a page with a high bounce rate, low conversions or where people leave a product in the shopping cart.

2. Define the goal

Next, define the goal of the A/B test—for example, a higher conversion rate, more clicks or sign-ups. Also, remember to set up a hypothesis stating what your expectation is.

3. Choose what you want to test

Choose what you want to test. An entire page? A CTA button? Or perhaps the position of a form. Ensure you only test 1 element, for example, a colour or a text. After all, if you test both, you still don’t know whether the colour or the text ensures better results.

4. Create 2 variants

As a next step, you create 2 variants based on your hypothesis. You don’t change anything about version A, the original version. Otherwise, you won’t know whether the current version works better than the test version, version B.

5. Determine statistical significance

Consider statistical significance. Statistical significance measures whether a result is meaningful or due to chance. Are the test results statistically significant? If so, you can see the results as truth and not created by chance.

6. Divide traffic

Use an A/B testing tool to divide traffic into 2 random groups. Ensure each version receives similar visitors so that the final data is correct.

Choose the target audience

Also, choose which target audience you will use. All traffic? Only new visitors? Or visitors from a particular region? Only create a segment if you receive enough traffic on your website.

7. Start the test

It’s time to start the A/B test. Check that the test is working correctly so you collect data. Or that is what the A/B testing tool does for you.

Tip: Take holidays and vacation periods into account. Both may affect test results.

8. Monitor the test

Once the test runs, it is essential to monitor it regularly. This is because version B may perform much worse than the original version. Since half of the traffic goes to version B, this could impact your business goals.

Tip: Run the test long enough to collect reliable data.

Implementing A/B test results

After the A/B test is complete, you can view the test results. Go to your A/B testing tool and check the result. The tool will provide insight into how both versions performed and the difference. Does version B work better than version A? In this case, you can implement version B on your website. Are the results the other way around? Leave version A for what it is.

Keep SEO in mind

An A/B test can affect your website’s SEO, so Google has created some tips:

• No cloaking

With cloaking, you show visitors different content than the search engine. And that is against the rules. So do not use an A/B test to mislead a search engine. Otherwise, you could be removed from the Google Index.

• Use rel=”canonical”

Set rel=”canonical” on version B that links to the original version A. This prevents the search engine from being confused by the multiple versions of 1 page.

• Set up a 302 redirect

Use a temporary 302 redirect on version B instead of a permanent 301 redirect so the search engine knows the redirect is temporary, and the URL of the original page should remain indexed.

A/B testing best practices

To make sure you run your A/B test in the best possible way, here are some A/B testing best practices.

1. Test only 1 element at a time. Otherwise, you won’t know which part had an impact.
2. Test both versions at the same time. Very different results may emerge a month later.
3. Run the A/B test long enough. This will help you collect reliable data.
4. Ask for feedback from visitors. This will give you more insight into what they find (un)pleasant.
5. Schedule the next A/B test. There is always room for more improvements.
6. Take into account external factors that may influence it. Consider trends, news, holidays and holiday periods.
7. Ensure sufficient traffic. Little traffic can lead to wrong data.

A summary

This article was about the question: what is A/B testing? Among other things, you learned how an A/B test works, why it is wise to take an A/B test and what all you can test. You have also been given a useful step-by-step plan that you can use to set up an A/B test yourself. Finally, it is essential to bear in mind that A/B testing should not affect the SEO of your website. We hope this article has helped you. Good luck!