A/B Testing is a way to compare up to 10 message variants for notifications in order to determine which one performs better, and is currently available for Push and Email notifications. A/B testing is often used to validate the effectiveness of message content, for example, to understand whether a notification with an image performs better than a text-only notification.

OneSignal allows you to experiment with up to ten variants on Pro and Enterprise plans. All other plans include testing two variants only for each notification.

Create Your Push Notification A/B Test

To run an A/B test, you need to

1. Create a New Push Message (Dashboard --> Messages --> New Push).
2. Select the target audience for the entire message. For example, players who completed level 10 of your game.
3. Add multiple variants for the same message by clicking on A/B Test.

4. Fill in the message content for each of the variants
5. Select the percentage of the target audience (out of the total from #2) that should receive the AB test messages.

6. Clicking on Review and Send will show you the preview panel with the key differences between the variants.
7. Messages that are sent as AB test will be marked as such under Dashboard --> Messages.

Example
Let's say the total count of players (Target Audience from point #2 above) who completed level 10 is 100 and you created an AB test message with two variants A and B (from point #3 above).

If you select 20% (point #5 above) as the AB test sample size (that is 20 players out of the total 100), then 10 players will get Message A and 10 players will get Message B.

📘

Best Practices: Selecting Target Audience Percentage

By default, the percentage of the target audience to receive the message is set to 25%, however, you may want to change this number depending on the size of the total audience.

For an A/B test to be effective, you must ensure that the percentage of the target audience selected contains enough users that the results cannot be easily skewed. Note also that the more variants you use, the larger your target audience percentage will need to be to ensure that a suitable number of users receive each variant.

While you may be tempted to set the audience percentage to 100%, be aware that this will mean that there will be no users from this segment who have not received a variant who can then receive the "winning" variant.

AB Test Report

Once you schedule an AB Test, you can view the performance details of the different variants under Dashboard --> AB Tests. The performance statistics on Clicks, Clickthrough Rate, and Delivered metrics are provided for all the variants. You could use these to determine the winner and send that winning message variant to the rest of your target audience.

You can also access the delivery report for individual variants from the Delivery tab.

Selecting A Winner

OneSignal provides three statistics for you to view the performance (Clicks, Clickthrough Rate, and Delivered) and choose a winner. Below screenshot describes how to view different stats and select a winning variant. We will then send the winner variant to all the remaining members of your target audience.

What you can A/B test

The OneSignal dashboard allows testing of different message contents as well as different notification options, such as images, sounds, action buttons, etc.

OneSignal does not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.