Dashboard > >
A/B Testing is a way to compare two notifications to determine which performs better. Marketers often use A/B testing to test different variations of the same message, or whether a notification with an image performs better than a text-only notification.
A/B tests are not yet available in our email beta. Read more
To run an A/B test, you must first createto test. Once you've done this, click New A/B Test in and do the following:
Name your A/B test - be sure to name it something memorable (e.g. "New Years headline test") that you can refer to later when reviewing your tests.
Select an audience that you want the A/B test to go to.
Modify contents of each message.
Schedule the test.
Dashboard > > >
Once you run an A/B test, you should get results something like this:
"There's a sale on new sheets"
"20% discount off new sheets"
We are 77% confident test B is better
The OneSignal dashboard allows testing of different message contents as well as different notification options, such as images, sounds, action buttons, etc.
OneSignal does not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.
Creating an A/B test works almost identically to, where you specify a set of users to send to, enter your message contents, and enter your delivery options.
The main difference is that in an A/B test, you must either select a template to send, or create your own. Most frequently, customers use A/B tests to test slight variants of message contents (in the example above).
Once you see which notification performs best, you can send it to the rest of the users. If a user was part of the A or B test group, they will not get the notification again.
So at first, only some of your users will get the notifications, but once you see how well the 2 different notifications perform, you can choose the "better" one to send to the rest of your users.
A/B tests do not send to all users in target segment(s) - they send only enough messages to reach statistical significance. You will need to go back to the A/B test results to choose which variant to send to the remainder of users.
A/B testing works by choosing segments and sending 2 different notifications to 2 small random groups of those segmented users. Our system creates these A and B group of random users between 5-25% of the total users in the targeted segments. The more users you have, the lower the percentage is.
The sample set needs to be large enough so 25% of your segment has 8,000 users or less. After that it will go down from there. If the segment has 20,000 it will be send to 10%.
You are not able to choose the percentage nor which users get added to the A/B groups.
Updated about a year ago