Run a simple A/B testing on an email campaign

Probance One lets you set up A/B tests quickly and easily.
You can create up to four versions of an automated campaign or newsletter to test different elements, such as the template, content personalization, sending time, subject line, or sender name. These A/B tests can be either simple or optimized.

A simple A/B test does not automatically select the best-performing version:

• For an automated campaign, you can monitor the test results and manually stop the test to select the version you want to keep.
• For a newsletter, all tested versions are sent to the full audience with no automatic selection.

This tutorial explains how to create a simple A/B test for an automated campaign or a newsletter.

1. Before you begin

Here are some things to know and consider before creating your A/B test:

• A/B tests are set up on existing campaigns and newsletters.
• When creating an A/B test, the versions are generated based on all the elements of the original campaign.
• If the A/B test is created from an empty campaign, you’ll need to configure each version manually. To save time, we recommend creating the A/B test after setting up the common elements of the campaign.
• The Custom sending time option is not available for campaigns using A/B testing. Conversely, if that option is enabled, you won’t be able to add an A/B test to the campaign.
• You cannot create an A/B test on follow-ups, or on campaigns that include follow-ups.

2. Creating an A/B test

1. On your campaign configuration page, scroll down to the A/B test block and click Create.
2. Once the block is updated, click Edit.
3. Check the Simple option.
4. In the Number of versions dropdown, select your test configuration. For example, to create 3 versions, select A/B/C.
5. Click Save to confirm.

3. Defining the variations

The Change section lets you define your variations by modifying the element(s) you want to test in each version. You can change the template, subject line, personalized offer, and sending time.

To create a variation, follow these steps:

1. Open the tab corresponding to the element you want to test.
2. Each tab allows you to configure each test version individually: click Edit under Campaign A to modify version A.
3. Enter the test parameters in the relevant fields, then click Save to confirm.
4. Repeat these steps for each version of your A/B test.

Tip: To get reliable conclusions from an A/B test, it’s best to vary only one element at a time. See our guide on A/B testing best practices to make the most of this feature.

4. Testing your versions

We highly recommend testing each version before activating your A/B test, to check the content and display across all configurations.

1. Go to the Template tab.
2. To test version A, click Send a test below the Campaign A preview.
3. In the Recipient list field, enter the email addresses for the test (one per line).
4. Customize the test (if needed):
   • If the template or subject includes personalization fields, use the Profile personalization section to define values.
   • If the email includes offers, set the number of offers and the selection mode to test different display configurations.
   • To replicate real sending conditions, enable the Restrict to campaign filters option. This applies the filters defined in the Personalized Offer section of the tested version.
5. Click Send to send the test email.

Repeat this process for each version of your A/B test.

5. Adding notes to your A/B test

You can add notes to your A/B test to track changes and describe the tested variations.

In the A/B test configuration block, click Notes.

Enter your notes and click Save and close to confirm.

These notes can be edited at any time, even while the A/B test is running or after it has ended.

6. Starting the sends

6.1. For an automated campaign

To start your A/B test sends, simply activate the campaign.

Each recipient will randomly receive one of the test versions, until you decide to stop the test.

Note: You can stop the A/B test at any time by deactivating the campaign.

6.2. For a newsletter

1. In the Timing section, click Edit.
2. Enter the desired Date and Time for sending.
3. Click Save to confirm.
4. Then, switch your newsletter to Active status.

Your A/B test will be sent at the defined time. Each recipient will randomly receive one of the test versions.

7. Viewing the results

You can monitor the results of your A/B test by clicking the Performance icon in the campaign details.

You can also access it via the Performance > A/B Tests menu and find your campaign in the list.

By default, the A/B test performance page shows the overall campaign results. You can view each version’s detailed results by selecting it in the carousel.

8. Stopping the test and selecting the best version

This step only applies to automated campaigns.

To stop your A/B test, follow these steps:

1. On your campaign configuration page, go to the A/B test block and click Delete.
2. Select the version you want to keep.
3. Click Apply.

The A/B test will be removed from the campaign, which will now use the selected version’s template, subject, and personalized offer.

The test results will remain available in the Performance > A/B Tests menu.