Run optimized A/B testing on an email-campaign

Probance One allows you to set up A/B tests quickly and easily. You can create up to four versions of an automated campaign or a newsletter to test different elements, such as the template, content personalization, send time, subject line, or sender name. These A/B tests can be either simple or optimized.

An optimized A/B test is fully automated: multiple variations of the same campaign are sent over a defined period— a few hours for a newsletter, several days or weeks for an automated campaign. At the end of this period, P-ONE automatically selects the best-performing version, which is then sent to the remaining recipients and to all future recipients of the campaign.

This tutorial will guide you through creating an optimized A/B test for an automated campaign or a newsletter.

1. Before you begin

Here are some important details to know and consider before creating your A/B test.

• A/B tests are configured on existing campaigns and newsletters.
• When creating an A/B test, the versions are generated by taking all elements from the base campaign.
• If the A/B test is created from a blank campaign, you will need to manually configure each element for each version. To save time, it is recommended to set up the A/B test after defining the common elements for all campaign versions.
• The Personalized send time option is not available for campaigns with an A/B test. Conversely, if this option is enabled on a campaign, you will not be able to configure an A/B test for it.
• You cannot create A/B tests on follow-up campaigns, nor on campaigns with follow-ups.

2. Create an A/B test

2.1. On an automated campaign

1. In the campaign configuration page, go to the A/B test section (at the bottom of the page) and click Create.
2. Once the section is updated, click Edit.
3. Check the Optimized option.
4. In the Number of versions dropdown, select the configuration for your A/B test. For example, to set 3 versions, select A/B/C.
5. In Best version sent, select the time frame for selecting the best-performing version, after which the A/B test will end. Two options are available:
   • As soon as possible: The best-performing version will be automatically selected by PONE when enough data has been collected for the results to be considered reliable.
   • No later than: The best-performing version will be automatically selected by PONE no later than the date you set.
6. In the Optimization criteria dropdown, select the data to be used to determine the best-performing version.
7. Click Save to confirm.

2.2. On a newsletter

1. In the newsletter configuration page, go to the A/B test section (at the bottom of the page) and click Create.
2. Once the section is updated, click Edit.
3. Check the Optimized option.
4. In the Total test volume dropdown, select what portion of the target audience should receive the tested versions. Two options are available:
   • 5% per test: Each version will be sent to 5% of the newsletter target (5% will receive version A, another 5% will receive version B, and so on if there are more than two versions). The rest of the target audience will receive the best version, automatically selected by Probance One.
   • 10% per test: Each tested version will be sent to 10% of the target audience (10% will receive version A, another 10% will receive version B, and so on if there are more than two versions). The rest of the target audience will receive the best version, automatically selected by Probance One.
5. In the Number of versions dropdown, select the configuration for your A/B test. For example, to set 3 versions, select A/B/C.
6. In Best version sent, select the timing for sending the best-performing version to the remaining target audience. Two options are available:
   • As soon as possible: As soon as the best-performing version is determined (results analyzed two hours after the A/B test is sent), it will be sent to the rest of the target audience.
   • The next day at the same time.
7. In the Optimization criteria dropdown, select the data to be used to determine the best-performing version.
8. Click Save to confirm.

3. Define the variants

The Change section allows you to define your variants by modifying the element(s) you wish to test for each version. You can vary the template, subject, personalized offer, and send time.

To create a variant, follow these steps:

1. Open the tab corresponding to the feature you wish to test.
2. These tabs allow you to configure each version of the test individually: click Edit on Campaign A to modify the content of version A.
3. Enter the parameters to be tested in the corresponding fields, then click Save to confirm.
4. Repeat these steps for each of the other versions of your A/B test.

To draw reliable conclusions from an A/B test, it is recommended to vary only one element at a time. Refer to our dedicated A/B testing best practices guide to optimize your use of this feature.

4. Test the versions

We highly recommend testing each version before activating your A/B test to verify the content and display of your email in all possible configurations.

Follow these steps to test your version:

1. Go to the Template tab.
2. Test Version A: To test version A, click the Send Test button located under the preview of Campaign A.
3. Add test recipients: In the Recipients list field, enter the email addresses that will receive the test, separating them with a line break.
4. Personalize the test (if needed):
   • If the template or subject of version A contains personalization fields, use the Profile personalization section to define the values to apply.
   • If the email includes offers, define the number of offers to integrate and the selection mode to test different display configurations.

Note: For a test that closely reflects the real sending conditions, activate the option Restrict to campaign filters. This option applies the constraints defined in the Personalized Offer section of the tested version for product selection.

5. Send the test: Click Send to dispatch the test email.

Repeat these steps for each version of your A/B test.

5. Annotate the A/B test

You can add notes to your A/B test. This allows you to keep track of modifications made to your test and describe the different variants being tested.

In the A/B test configuration block, click on Notes.

Enter your notes and click Save and Close to confirm.

These notes can be edited at any time, including during the execution of the A/B test and once it has been completed.

6. Start the sendings

6.1. On an automated campaign

Once your variants are configured, you can activate your campaign to start sending.

Each recipient of the campaign will receive one of the tested versions, selected randomly, until P-ONE selects the best version.

Note: You can interrupt your A/B test at any time by deactivating the campaign.

6.2. On a newsletter

1. In the Timing section, click Edit.
2. Enter the desired Sendind date and time.
3. Click Save to confirm.
4. Then, set your newsletter to Active status.

Your A/B test will be sent at the defined Timing.

7. View the A/B test results

If your A/B test is still ongoing or pertains to a newsletter, you can view its results by clicking on the Campaign Performance icon.

If the A/B test is finished, you can access it via the Performance > A/B Tests menu by searching for your campaign in the list.

By default, the A/B test performance page will display the overall results of the campaign. You can view the detailed results of each version by selecting it from the carousel.