In an A/B test you set up two variations of the one campaign and send them to a small percentage of your total recipients. Half of the test group is sent Version A, while the other half gets Version B. The result, measured by the most opens or clicks, determines the winning version. This is then sent to the remaining subscribers.

When an A/B test is in progress you can't change the campaign in any way, so be sure to test thoroughly before you start.

You can A/B test an email campaign in one of three ways. To get started:

The three test types are explained below.

Subject line

For this test, campaign version A and B are identical except for the subject line. For example, you could test to see if a generic subject gets more opens than a longer subject line that's more specific.

Alternatively, you could:

From name

Sender details are important because many people will not open an email if they don't recognize who it's from. With the From name test you can use a different name andemail address for Version A versus Version B, as shown below, or just change one or the other.

The best approach depends on your relationship to the subscriber. Consider if they are more likely to recognize an individual's name, your company name, or the product name your campaigns are about.

Email content

This is to test different elements of the campaign itself, for example: section titles, article length, calls-to-action, header images and more. You might even test two completely different designs to see which one gets the most clicks.

Campaigns for this type of test can be created using one of your saved templates or designed externally and imported (if that option is available to you). If you're doing an email content test, follow the on-screen instructions to set up Version A first. After that, you'll be prompted to repeat the process to set up Version B.

If you're using the same template and only making minor changes to content, save time by copying the content from Version A to create Version B. You can do so by selecting your version A content from the top left corner of the template library in Recent emails, as shown in the image below:

When both versions of the campaign are set up you'll be prompted to choose the subscriber list, or lists, to send to. If new subscribers are added to a list after the A/B test has started they will not be sent the campaign. Any subscribers added while a test is in progress will have to be targeted separately.

The next step is selecting the size of your test group, deciding how the winner will be chosen, and setting a length of time to run the test.

Test group size

Use the slider to define a test group, which should be small subset of your recipients, say 20–30%. Recipients in the test group are selected randomly. Half of them will be sent Version A while the other half are sent Version B. The remaining recipients will be sent the winning version.

When you start a test we send all emails (version A and B) to the test group at the same time. When the winner is decided, the winning version is sent to all remaining recipients at the same time.

How will the winner be decided?

Select either Open rate, Total unique clicks or Total clicks on a selected link as the performance metric for your test. You can't change the metric after a test has started but you can monitor activity for all three metrics while the test is in progress.

Test run time

The length of time you can run an A/B test is from one hour, up to five days. There's also the option to manually select a winner while the test is in progress.

It's very important to test and check for errors of any kind before you start an A/B test because you cannot make any changes when an A/B test is in progress.

To double-check everything you've set up so far, we summarize it for you in a campaign snapshot. If you need to change something just click the the Edit buttons on the right of the snapshot.

If you're testing email content using a template-based campaign, the snapshot is where you can check the plain text version of your emails. What looks great as HTML might need a little work as text only. Click Preview to see how they look and edit to make changes, as highlighted above.

When you're satisfied that everything is good to go click Test and define delivery. Run a Quick test to send yourself previews of both versions or click Full test to run a fully automated design and spam test.

Send now or schedule for later

You can start the A/B test immediately or select Schedule it to start at the following time.

If you schedule the test to start later, we'll send you an email notification when it starts.

While a test is in progress you can monitor results with the A/B test report, which looks like this:

The graph makes it easy to see which version of the campaign is winning, according to the performance metric you've chosen: open rate, total unique clicks, or total clicks on a selected link. Below the graph you can also see which version is winning for the two metrics you didn't select.

For some A/B tests there's a clear winner before the test time is up. On the report page, pictured above, you can see the time remaining and a link to manually select the winner. Click this to choose the winner yourself and end the test early.

After a winner has been decided, through test completion or manual selection, the remaining recipients will be sent the winning version and a full campaign report will become available. The A/B test results are included in the full report so you can revisit them when designing your next campaign.

If you let the test run to completion we'll send you an email notification as soon as it's finished. If the two versions are tied at the end of the test, version A will be sent to the rest of the list.