Today, A/B testing is essential for a successful emailing strategy.
If you want to improve the results of your automated scenarios and newsletters, you have two options: you can trust your intuition or you implement tests.
What is A/B testing ?
This method consists in testing two versions of a web page or an email, in order to identify which version will perform best according to the defined objective (number of visitors, clicks, registrations, openings, bounce rate…). This will improve the results of your marketing campaigns. You then analyze how the audience engage and perform with each version to determine which is optimal for you to implement.
You can conduct several different tests on the same device, but it is imperative that you test one item at a time. If you test multiple items at the same time, you take the risk of not being able to identify which one had an impact on the results.
A/B testing is the most effective way to improve the performance of your emails.
Why? Because you will be able to use solid data directly from your audience. You can focus on your strategic choices and on the elements that have a real impact on your results.
Why conduct A/B Tests ?
A/B testing will help you to :
- Build and test hypotheses by identifying whether the changes you made have an impact.
- Optimize the user experience by understanding how elements can influence user behavior.
- Capitalize on the best performing elements to improve your marketing campaigns.
As you can see, we no longer say “I think that…” but “I test and analyze !” From now on, you will make decisions based on data and not on your intuition.
The goal is to simplify your decision making and allow you to create more effective email campaigns for your audience. These tests should allow you to improve your open and click rates. As a result, these tests will significantly increase lead acquisition, sales and revenue.
How to conduct A/B Tests ?
1: Define your goal
First, you need to determine what you want to achieve with your A/B test (increase your email open rates, click-through rates?) and think about the changes you need to make to achieve the desired results. Testing without a specific goal is a waste of time.
For each test, describe your starting premise, the hypothesis and the indicators you are tracking.
For example, I observe that half of the people who receive my newsletters do not click on the links on my blog (hypothesis). I will put orange instead of grey on the “Read article” buttons, the click rate should increase (hypothesis). I want to get 5% more clicks on the “Read article” buttons (goal).
2: Focus on the emails you send regularly
Initially, focus on newsletters/scenarios that are sent frequently in order to have a high volume of mailings.
To get reliable results, you need to have a large enough sample size. To send a relevant A/B test campaign, you will need to test your two versions on a sample of at least 1000 contacts. Below this threshold, you run the risk of not collecting sufficiently conclusive data at the end of the test.
=> Therefore, if you have a database of 2,000 subscribers, and you send a newsletter every month, the expected results will take much longer to reveal themselves than with a database holding 200,000 subscribers to whom you send a newsletter every day.
Nevertheless, if your database contains several thousand contacts, we recommend that you segment a part of it to perform the test. You can then send the best performing version to the rest of the database.
3: Test one item at a time
It is recommended to test only one variable in order to be able to determine which one has an impact on the result of the experiment. Focus on one item at a time and do not change the other variables.
If you want to test different elements, you have to conduct a new experiment each time.
4: Check if your samples are statistically significant
You can choose to run an A/B test on a part of your database or on the whole database, as long as you have a significant sample. A good practice is to choose the same sample size.
This will allow you to compare the results on a similar volume of data and gather statistics that are representative of reality.
For example, you can simply divide your database in two, 50% version A (existing newsletter) and 50% version B (newsletter with changes). You can also choose to send version A to 10% of your sample, version B (with change) to 10% of your sample and so on if you have versions C and D. The rest of the target group will receive the remainder, that is, the best version of the test according to your objective.
To learn more about setting up your A/B test, please read our articles How to create an A/B test campaign and How to create an A/B test on a newsletter.
5: Follow and analyze the conducted tests
You need to track the results of your tests and analyze them to improve the performance of your scenarios and newsletters. You can then adapt your strategy according to the results.
We are often asked : How long does a test last?
We can answer 2 days on average after the sending but the reality is that there is no common duration for everyone.
As explained in rule n°2, try to do your A/B tests on scenarios or newsletters that have high volumes and/or are sent frequently.
Be careful, very often a single A/B test is not enough to meet the objective, so it will have to be repeated several times.
Following an A/B test, there are two possible scenarios:
– Either it was conclusive, that is to say that a variable answered or not the hypothesis. In this case, we advise you to launch another A/B test to confirm the hypothesis.
– Either, no variable has answered the hypothesis, you will have to start the test again until you reach your objective.
Our advice: do not perform your A/B tests during a peak period in your sector. Indeed, external factors may distort the results.
For example, if you have an online wine sales site, avoid doing a test during wine fairs.
6: Test all the time
A/B testing is a continuous optimization process. Each test will allow you to draw conclusions and guide you to new hypotheses to test. Once you know which subject line, for example, is the best for your audience, you can select a new element and conduct a new test.
What to test ?
First, segment the variables to be tested:
- The elements inherent to the sending such as the sender address, the subject line and the pre-header.
- The elements that make up the body of the email, i.e. the layout of the text and images, the trigger buttons, etc.
Depending on your KPI’s, and your emailing strategy, you will choose to improve your open rates or your click rates or maybe both?
To improve your open rate
It is important to test the elements related to the sending because they are the first elements visible by your audience, even before the message is opened. Upon receiving a newsletter, a person will decide in a few seconds if they will open the email or if they will put it in the trash. Following these tests, you will be able to identify what works best with your target.
- For the sender’s name, you can for example test putting your brand name on one version and a first name with the brand name on the other version, if you wonder if your audience is more receptive to a human touch.
- For the email subject line, you can try with or without personalization, write text alone or with smileys. You could also test clear message “Check out our selection for Mother’s Day” or a teasing one “A selection just for you…”
To improve your click rates
An important element to insert on your newsletters is the call to action button, indeed, it is used to make your audience click on the offer, the product…and thus increase your conversion rate. You can test its positioning, its color, the message…
The text and the offer put forward are important factors to make people want to click but there are also the visuals used!
Don’t hesitate to test different images, pictures or GIFs?
You can for example, test the writing of a personalized or rather product-oriented content versus more editorial content. Depending on the results, this will give you an indication of what appeals to your audience and help you review your marketing strategy.
To conclude ?
A/B testing has become an essential method to improve your results, your return on investment and the user experience. Before you start, define your objective, your hypothesis and most importantly track and analyze your results! Finally, don’t forget the golden rules ?
Now it’s your turn!