A/B Testing

A/B testing is a tool to help you test different versions of a message to understand its effectiveness. It provides insights to improve engagement and help you achieve your objectives. With A/B testing, you will get insights on which designs or language result in a higher open and clickthrough rate. This is a particularly useful tool for Marketers, especially for Lifecycle and Growth.

Insights from A/B testing can be used across the company to reach broader business goals. You can also more confidently improve the performance of your messages.

For example, A/B testing can allow you to understand whether a notification with an image performs better than a text-only notification.

With OneSignal, you will be able to A/B test up to 10 message variants to determine which one performs better.

We enable experimentation with up to ten variants on Pro and Enterprise plans. All other plans include testing two variants only for each notification.

Note: A/B testing capabilities are currently available for Push and Email notifications.

Create Your Push Notification A/B Test

To run an A/B test, you need to

1. Create a New Push Message (Dashboard --> Messages --> New Push).
2. Select the target audience for the entire message. For example, players who completed level 10 of your game.
3. Add multiple variants for the same message by clicking on A/B Test.

4. Fill in the message content for each of the variants
5. Select the percentage of the target audience (out of the total from #2) that should receive the AB test messages.

6. Clicking on Review and Send will show you the preview panel with the key differences between the variants.
7. Messages that are sent as AB test will be marked as such under Dashboard --> Messages.

Example
Let's say the total count of players (Target Audience from point #2 above) who completed level 10 is 100 and you created an AB test message with two variants A and B (from point #3 above).

If you select 20% (point #5 above) as the AB test sample size (that is 20 players out of the total 100), then 10 players will get Message A and 10 players will get Message B.

A/B Test Report

Once you schedule an A/BTest, you can view the performance details of the different variants under Dashboard --> A/B Tests. The performance statistics on Clicks, Clickthrough Rate, and Delivered metrics are provided for all the variants. You could use these to determine the winner and send that winning message variant to the rest of your target audience.

You can also access the delivery report for individual variants from the Delivery tab.

Selecting A Winner

OneSignal provides three statistics for you to view the performance (Clicks, Clickthrough Rate, and Delivered) and choose a winner. The screenshot below describes how to view different stats and select a winning variant. We will then send the winner variant to all the remaining members of your target audience.

What you can A/B test

The OneSignal dashboard allows testing of different message contents as well as different notification options, such as images, sounds, action buttons, etc.

OneSignal does not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.