Email A/B Testing

Email A/B testing is a tool to help you test different versions of emails. It enables you to understand email effectiveness, provide insights to improve engagement and help you achieve your objectives. With A/B testing, you will get insights on what language or design results in a higher open and clickthrough rate. This is particularly useful for marketers, especially for Lifecycle and Growth.

Insights from A/B testing can be used across the company to reach broader business goals. You can also more confidently improve email performance.

OneSignal is the only platform that provides free Email A/B testing that is intuitive and easy to use. The feature is available for all pricing plans, however Free and Growth plan users are limited to having two variants per email. Access to experimenting with up to 10 variants requires a Pro or Enterprise plan.

What You Can A/B Test

The OneSignal dashboard allows the testing of different message contents.

Creating Your Email A/B Test

Getting Started

1. Select New Email (Dashboard > Messages > Email). A/B Testing works across all email editors. Select your editor experience.

2. Start creating an email. You’ll need to indicate:

• Message Name
• Included and excluded segments
• Sender name and email
• Email subject
• Select ”[Track link clicks] (https://documentation.onesignal.com/docs/email-links-and-click-tracking)” if you want to enable link tracking.
  Note that when checked, if you have link tracking enabled in your email service provider, your links will be converted and tracked. When checked, if you have link tracking enabled in your email service provider, your links will be converted and tracked. Turning this off will ensure your links are not changed or tracked.
• Email content (with an option to create a template)

Step 1: Add Variants

Click on the A/B test button on the top right side of the email preview to create another variant. A tab above the Subject in the Message section will appear and a duplicate of your first variant will be in the editor. You may create more variants, which will duplicate the preceding version. If you need to duplicate the first variant, make sure you’re on that tab before clicking “Add Variant”.

Step 2: Create Variants

Create the different variants. Changes made to a specific variant will not affect the other variants.

Step 3: Select How Many Users Receive the Variants

Select the percentage of your audience that should receive the A/B test messages. (i.e. 20% means 20% of the segment selected will randomly receive one of the variants.)

Step 4: Schedule the A/B Test

Select whether to send the email variants immediately or at a specific time.

Step 5: Review and Send

Clicking on Review and Send will show you the preview panel with the key differences between the variants.

Step 6: Review Results

Messages that are sent as A/B tests will be marked as such under Dashboard —> Messages —> Email —> A/B Tests Tab. You can click into the report for each test to see the full report or view the variant-specific reports.

Step 7: Select a Winner

We provide some statistics for you to view the performance and choose a winner. The below screenshot describes how to view different stats and select a winning variant. We will then send the winner variant to all the remaining members of your target audience.

Example
Let's say the total count of players who completed level 10 is 100 and you created an A/B test message with two variants A and B.

If you select 20% as the A/B test sample size (that is 20 players out of the total 100), then 10 players will get Message A and 10 players will get Message B.

A/B Test Report

Once you schedule an A/B Test, you can view the performance details of the different variants under Messages —> Email —> A/B Test. The performance statistics on Open Rate, Clicks, Clickthrough Rate, and Delivered metrics are provided for all the variants. You can use these to determine the winner and send that winning message variant to the rest of your target audience.

You can also access the delivery report for individual variants from the delivery summary (Messages → Email —> click on the email) and for the whole experiment in the A/B tests tab (Messages —> Email —> A/B Test).

Tips

Understand the benchmarks

What do past results look like? This is important to understanding the results you obtain from A/B testing. This will also help you set a practical goal.

Have a clear goal and hypothesis

What results are you aiming to achieve? What do you think is going to happen? These guiding questions will allow you to build an effective A/B test.

Control your experiment

Have a control version of your email, which is the version you would have sent before testing any variables. Only change one variable at a time to get clear and helpful insights. To do this, finalize the first variant before creating all the other variants. Then go into each variant to change that one variable.

The results with your control are the baseline to measure the results from your variants.

Test the variants at the same time

Timing matters so you want to ensure users see the message at the same time and day.

Continuously experiment to optimize your strategy

There are many variables to test, such as:

• Subject lines
• Email layouts (e.g., Image vs. image with text vs. image and text)
• Different CTA types (e.g., “Start Trial” vs “Claim My Free Trial”)
• Email copy lengths
• Offers/promotions
• Landing pages

---

FAQ

Can I A/B test with different segments as the variable?

We do not support testing different segments, nor does it support testing different delivery and scheduling options. These limitations exist because A/B testing requires random samples of users in target segments delivered at the same time.

How do I auto-select a winner?

This is not currently available. You must manually select the winner.

How do I add a control segment?

A control segment is a group of users who don't get the message. Automatic creation of a control group is currently not available, but it is possible to set one up utilizing data tags and excluding the segment of users with this data tag from your messages.

Once you have a list of users you want as a control group, you can easily set a tag on them through the dashboard CSV Upload or API Edit tags with external user id.

Want a new feature? Leave us a note in the chat!