A/B Testing

📘

AB Testing has Updated!

If you are looking for the legacy AB Testing docs go here

What is A/B Testing?

A/B Testing is a way to compare up to 10 message variants on the same notifications to determine which one performs better. Marketers often use A/B testing to validate the effectiveness of a particular message content, for example, to understand whether a notification with an image performs better than a text-only notification.

OneSignal allows you to experiment with up to ten variants on Pro and Enterprise plans. All other plans include testing two variants only for each notification.

🚧

Push Only

A/B tests are not yet available for our Email and SMS products.

Create A/B Test

To run an A/B test, you need to

1. Create a New Push Message (Dashboard --> Messages --> New Push).
2. Select the target audience for the entire message. For example, players who completed level 10 of your game.
3. Add multiple variants for the same message by clicking on A/B Test.

4. Fill in the message content for each of the variants
5. Select the percentage of the target audience (out of the total from #2) that should receive the AB test messages.

6. Clicking on Review and Send will show you the preview panel with the key differences between the variants.
7. Messages that are sent as AB test will be marked as such under Dashboard --> Messages.

Example
Let's say the total count of players (Target Audience from point #2 above) who completed level 10 is 100 and you created an AB test message with two variants A and B (from point #3 above).

If you select 20% (point #5 above) as the AB test sample size (that is 20 players out of the total 100), then 10 players will get Message A and 10 players will get Message B.

AB Test Report

Once you schedule an AB Test, you can view the performance details of the different variants under Dashboard --> AB Tests. The performance statistics on Clicks, Clickthrough Rate, and Delivered metrics are provided for all the variants. You could use these to determine the winner and send that winning message variant to the rest of your target audience.

You can also access the delivery report for individual variants from the Delivery tab.

Selecting A Winner

OneSignal provides three statistics for you to view the performance (Clicks, Clickthrough Rate, and Delivered) and choose a winner. Below screenshot describes how to view different stats and select a winning variant. We will send the winner variant to all the remaining members of your target audience.

What you can A/B test

The OneSignal dashboard allows testing of different push message contents as well as different notification options, such as images, sounds, action buttons, etc.

OneSignal does not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.