Dashboard > Messages > A/B Tests

A/B Testing is a way to compare two notifications to determine which performs better. Marketers often use A/B testing to test different variations of the same message, or whether a notification with an image performs better than a text-only notification.

🚧

A/B tests are not yet available in our email beta. Read more

Create A/B Test

To run an A/B test, you must first create Templates to test. Once you've done this, click New A/B Test in A/B Tests and do the following:

1. Name your A/B test - be sure to name it something memorable (e.g. "New Years headline test") that you can refer to later when reviewing your tests.

2. Select an audience that you want the A/B test to go to.

3. Select Templates to test.

4. Modify contents of each message.

5. Schedule the test.

Example A/B Test Result

Dashboard > Messages > A/B Tests > A/B Test Report

Once you run an A/B test, you should get results something like this:

What you can A/B test

The OneSignal dashboard allows testing of different message contents as well as different notification options, such as images, sounds, action buttons, etc.

OneSignal does not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.

Running an A/B test

Creating an A/B test works almost identically to New Message, where you specify a set of users to send to, enter your message contents, and enter your delivery options.

The main difference is that in an A/B test, you must either select a template to send, or create your own. Most frequently, customers use A/B tests to test slight variants of message contents (in the example above).

Once you see which notification performs best, you can send it to the rest of the users. If a user was part of the A or B test group, they will not get the notification again.

So at first, only some of your users will get the notifications, but once you see how well the 2 different notifications perform, you can choose the "better" one to send to the rest of your users.

🚧

Action Required

A/B tests do not send to all users in target segment(s) - they send only enough messages to reach statistical significance. You will need to go back to the A/B test results to choose which variant to send to the remainder of users.

How A/B test groups work

A/B testing works by choosing segments and sending 2 different notifications to 2 small random groups of those segmented users. Our system creates these A and B group of random users between 5-25% of the total users in the targeted segments. The more users you have, the lower the percentage is.

The sample set needs to be large enough so 25% of your segment has 8,000 users or less. After that it will go down from there. If the segment has 20,000 it will be send to 10%.

You are not able to choose the percentage nor which users get added to the A/B groups.