Create an optimized A/B test on an SMS campaign

Probance One allows you to set up A/B tests quickly and easily. You can create up to four versions of an SMS to test different elements, such as message content, personalized offers, send time, or sender name. These A/B tests can be simple or optimized.

An optimized A/B test is fully automated: several variants of the same campaign are sent over a defined period — a few hours for an ad-hoc campaign, several days or weeks for an automated campaign. At the end of this period, P-ONE automatically selects the best-performing version, which is then sent to the rest of the target audience and to all future campaign recipients.

This tutorial explains how to create an optimized A/B test for an SMS campaign.

1. Before you begin

Here is some information to know and consider before creating your A/B test.

• A/B tests are configured on existing campaigns.
• When creating an A/B test, versions are generated by copying all elements from the base campaign.
• If the A/B test is created from a blank campaign, you will need to manually configure each element for each version. To save time, it is recommended to set up the A/B test after defining the elements common to all versions of the campaign.
• You cannot create an A/B test on resends or on campaigns with resends.

2. Create an A/B test

2.1. For an automated campaign

1. On your campaign configuration page, go to the A/B test block (at the bottom of the page) and click Create.
2. Once the block has been updated, click Edit.
3. Check the Optimized option.
4. In the Number of versions dropdown list, select your A/B test configuration. For example, to define 3 versions, select A/B/C.
5. In Best version sent, select the deadline for selecting the best-performing version, after which the A/B test will end. Two options are available:
   • As soon as possible: the best-performing version will be automatically selected by PONE once enough data has been collected for the results to be considered conclusive.
   • No later than: the best-performing version will be automatically selected by PONE no later than the date you specify.
6. Click Save to confirm.

2.2. For a newsletter

1. On your campaign configuration page, go to the A/B test block (at the bottom of the page) and click Create.
2. Once the block has been updated, click Edit.
3. Check the Optimized option.
4. In the Total test volume dropdown list, select what proportion of the target audience should receive the tested versions. Two options are available:
   • 5% per test means that each version will be sent to 5% of the newsletter’s target audience (5% will receive version A, another 5% of the target will receive version B, and so on if there are more than two versions). The rest of the target audience will receive the best version, automatically selected by Probance One.
   • 10% per test means that each tested version will be sent to 10% of the target audience (10% will receive version A, another 10% of the target will receive version B, and so on if there are more than two versions). The rest of the target audience will receive the best version, automatically selected by Probance One.
5. In the Number of versions dropdown list, select your A/B test configuration. For example, to define 3 versions, select A/B/C.
6. In Best version sent, select the delay for sending the best-performing version to the remainder of the target audience. Two options are available:
   • As soon as possible: as soon as the best-performing version is determined (results analyzed two hours after the A/B test is sent), it is sent to the rest of the target audience.
   • The next day at the same time
7. Click Save to confirm.

3. Define the variants

The Change section allows you to define your variants by modifying the element(s) you want to test for each version. You can vary the template, the subject, the personalized offer, and the send time.

To create a variant, follow these steps.

1. Open the tab corresponding to the characteristic you want to test.
2. These tabs allow you to individually configure each test version: click Edit next to Campaign A to modify the content of version A.
3. Enter the parameters to test in the corresponding fields, then click Save to confirm.
4. Repeat these steps for each of the other versions of your A/B test.

To draw reliable conclusions from an A/B test, it is advisable to vary only one element at a time. Check out our dedicated guide to A/B testing best practices to optimize your use of the feature.

4. Test the versions

We strongly recommend testing each version before activating your A/B test, to verify the content of your SMS in all possible configurations.

1. Go to the Template tab.
2. Test version A: To test version A, click the Send a test button located below the Campaign A preview.
3. Add test recipients: in the Recipients list field, enter the phone numbers that will receive the test, separating them with a line break.
4. Customize the test (if needed):
   • If the message contains personalization fields, use the Profile personalization section to define the values to apply.
   • If the message includes offers, define the number of offers to include and the selection mode to test different display configurations.

Note: For a test that closely matches real send conditions, enable the Restrict to campaign filters option. This option applies the constraints defined in the Personalized offer section of the tested version for product selection.

5. Send the test: click Send to dispatch the test SMS.
6. Repeat these steps for each version of your A/B test.

5. Annotate the A/B test

You can add notes to your A/B test. This allows you to keep a written record of the changes made to your test and to describe the different variants tested.

In the A/B test configuration block, click Notes.

Enter your notes and click Save and close to confirm.

These notes can be modified at any time, including during the A/B test execution and after it is completed.

6. Start sending

6.1. For an automated campaign

Once your variants are configured, you can activate your campaign to start sending.

Each campaign recipient will receive one of the tested versions, randomly selected, until P-ONE selects the best version.

Note: You can stop your A/B test at any time by deactivating the campaign.

6.2. For a newsletter

1. In the Timing section, click Edit.
2. Enter the desired Date and Send time.
3. Click Save to confirm.
4. Then set your newsletter to Active status.

Your A/B test will be sent at the defined timing.

7. View the A/B test results

If your A/B test is still in progress, or is for a newsletter, you can view its results by clicking the Performance icon of the campaign.

If the A/B test is completed, you can access it through the Performance > A/B tests menu, by searching for your campaign in the list.

By default, the A/B test performance page displays overall campaign results. You can view the detailed results of each version by selecting it in the carousel.