A/B testing in email marketing

As email marketing is going through its renaissance era, a lot of people claim to be experts in the field. This creates plenty of chaos.

On one hand, it’s absolutely disastrous. On the other hand, it allows us to apply chaos theory to the realities of email marketing. The butterfly effect, in particular: a small change in one state of a deterministic nonlinear system can result in large differences in a later state.

In other words, email as a marketing asset is so intricate that even the tiniest change can make or break your entire campaign.

As we’re going through a post-pandemic crisis, the last thing you want to happen is for your email campaign to fail. That’s why you need to be particularly careful with all the changes you make. Ideally, you should back all your email marketing decisions with data.

One way to obtain that data is to A/B test your emails.

What Is A/B Testing In Email Marketing?

A/B testing, in the context of email, refers to the process of sending one variation of your email campaign to a part of your mailing list subscribers and a different variation to another part of subscribers.

The goal of A/B testing is to identify which element variation shows the best results and use it to craft the ultimate email campaign.

Why Do You Need to A/B Test Your Emails?

If you ever read any of our how-to guides on email marketing, you must have noticed that regardless of the campaign element we discuss in those, we always tell you to A/B test your emails. There’s a good reason for that.

A/B testing has a wide range of benefits that can make your email campaign significantly more effective:

• It helps to understand what your subscribers want.
  You need to remember that the end goal of every marketing asset you create is to appeal to the target audience. Even if you personally think that the email you’ve crafted is above and beyond, you can’t just send it out immediately and call it a day. It’s mandatory to get a second opinion from your subscribers first. The last word is always after the emails’ end-receivers in question. By letting your subscribers decide what works best for them, you increase your chances of proving tailor-made emails that fit your prospects.
• It allows maintaining a competitive advantage.
  A couple of years ago, A/B testing your emails wasn’t as common as it is today. Back in the day, doing so would give you a competitive advantage and differentiate you from other businesses in your industry. Today, it’s no longer an option but rather a necessity. All your competitors are A/B testing their emails, which means you need to match their efforts if you want to stay on par with them.
• It’s accurate.
  You can always use your logic when assessing how your email campaign’s different elements can impact your prospects. There’s plenty of information about the effects of varying email marketing elements on an email campaign’s success. However, there will always be bias in your assumptions. First of all, people tend to be less harsh towards the creations of their own. Secondly, you can never be sure that something that works well for your colleagues will have the same results for you. On the contrary, A/B testing provides you with hard facts and figures to base your further analytics on.
• It’s cost-effective and time-effective.
  Since you have access to all the information about your target audience’s response to your email marketing efforts, you can promptly discontinue those that don’t generate as much return on investment as expected. Equally, when A/B testing, you save yourself some time that would’ve otherwise been wasted on a strategy that doesn’t convert.
• It helps to boost open and click-through rates.
  A/B testing helps identify the common trends and preferences among your target audience and appeal to them more effectively. Knowing about what they like and dislike, you can craft an email that would consider that.
• It allows converting more prospects into customers.
  With higher open rates and CTR, you get a chance of generating more revenue as more people act upon the CTA in your email.

How to A/B Test an Email

In our previous articles, we’ve already covered the basics of A/B testing and talked about how to A/B test like a pro. You should definitely check out those as the information laid out there applies to email A/B testing, too.

Nonetheless, just like we always do at NetHunt, we still have plenty of tips and tricks to share with you. Especially since email marketing A/B tests have their own peculiarities that need to be discussed.

Before we get down to the actual testing part, it’s important to learn what options you have. After all, emails are very multi-faceted marketing assets with many variables and combinations thereof that can be put to the test.

1. Subject Lines

We know that it’s wrong to judge the book by its cover, but we simply can’t help ourselves. The book’s cover is the first thing you see at the store when trying to find your next read, so it’s only fair that you make up your initial judgment based entirely off it.

The same thing happens with emails. As a new email lands into your inbox, you don’t see the contents of it at first. The only thing visible to you at that point is the subject line. That’s why first impressions matter!

Your goal is to achieve the perfect balance between informative and intriguing, without click-baiting your way into the recipient’s read-list. If your email fails to deliver of what it promised, you will face the prosecutions aka going straight into spam.

69% of email recipients report email as spam based on subject lines alone.[ConvinceAndConvert]

A successful subject line is the backbone of your campaign. That’s why you need to work hard on getting it right - your entire campaign depends on it. It’s so important that we even dedicated a whole article to creating a good one. We’ve worked extra hard to collect all the information you need to craft a subject line that gets your email opened.

Make sure to give it a read!

Nonetheless, even after reading the article, you still can’t just implement all the knowledge you’ve gained into your email marketing campaign and get over it. You need to make sure that those tips and tricks work well for your particular audience. The only way to do that is to A/B test your subject lines.

Here are some data-backed subject line A/B testing ideas for you to try out:

1. Personalisation

According to Experian, emails with personalised subject lines are 26% more likely to be opened.

While modern personalisation goes far beyond merely incorporating the recipient’s name into the subject line or the email body, it’s still a good place to start. In fact, according to Campaign Monitor’s study on Power Words in Email Subject Lines, including the person’s name into the subject line can lead to a 14% increase in open rates.

Any variation of the name can bring your email campaign positive results. But you need to remember that the effectiveness of using just the first name or just the last name, or the combination of both varies from industry to industry. You need to try out both options and see which one works best for you.

2. Ambiguous benefits vs Specific benefits

The person’s name might be the sweetest sound to their ears, but the notion of potential benefits comes a close second!

One of the first rules of creating a subject line that encourages your email subscribers to click on the email is stating exactly how they will benefit from doing so. You need to provide a promise of value from the very first word. Otherwise, it’s a waste of their time, and they’re not going to open the message from you.

The trick is… Each is to their own. As an author of the email, you genuinely believe that all the content you create is worthy of attention and brings a lot of value to whoever interacts with it. The truth, however, can be somewhat different. Just because you think that something is best for your audience, it doesn’t mean they necessarily agree.

If your email campaign offers multiple benefits to the subscribers, you need to A/B test all of them to see which one has the most value. Spoiler alert: It’s not always the most obvious one.

A great example of why this subject line A/B test is essential is Optimizely’s email campaign. Back in 2014, Optimizely ran A/B tests to determine which subject line would generate the most clicks.

• Option A: Get Optimizely Certified for 50% Off
• Option B: Get Optimizely Certified & Advance Your Career

Option A assumed that the subscribers would be more interested in a monetary benefit, while Option B promised a better career. Usually, we would say that a more specific benefit is more effective, but this particular A/B test proved otherwise. It turned out that Option B, the challenger, had 13.3% more opens.

3. Subject line length

Brevity is the soul of wit. It’s always been that way, but even more so now.

People are steadily shifting towards using smartphones to check their emails instead of reading their morning mail on the desktop. In fact, roughly 85% of users these days use smartphones to access email. The two things associated with a change like that are:

• Display issues on different devices and browsers - limited space for the subject line.
• People skim through their inboxes and long subject lines dilute the main subject.

Today, the perfect approach is to keep the subject line short and sweet, with the ideal length being around 6-10 words:

Nonetheless, you see how much variation there is. This means you need to conduct your own A/B tests to determine which subject line length works best for you and your audience.

4. Emojis vs No emoji

There’s a whole debate of whether emojis are a hit or a miss in 2021. Some people claim that emojis are the lord and saviour of email marketing, while the others are a long time over the emoji hype and consider them to be outdated and tacky.  

Either way, you can’t deny the fact that emojis are a great roundabout for the subject line character limit (you can easily replace a whole emotion with a tiny emoji). Besides, they help your emails stand out in the inbox, too. Have a look at mine for proof:

Your eyes immediately wander off to the subject lines with emojis in it. So you can be sure that your email will be seen.

Another question is, will it be opened. Several years ago, the answer would have been a hard yes. Today, however, there is no evidence that emojis increase open rates by a lot (on average, a 2-3% variation).

However, it could be the case for your subscribers. Your audience might be more or less susceptible to emojis in general as well as to particular emojis:

To find out whether emojis are a good fit for your email marketing strategy, put them to an A/B test!

5. Urgency vs No urgency

One of the sneakiest yet most effective ways to improve your emails’ open rates is to leverage people’s fear of missing out.

According to Robert Cialdini, urgency and scarcity are the ‘drivers of influence’, which explains why playing on the FOMO is an effective email marketing technique.

According to a study done by Mailchimp, words like urgent, breaking, important and alert tend to improve open rates dramatically:

Similarly, phrases like ‘# of days left’ and ‘last chance’ tend to come across as more actionable for subscribers than others.

2. Copy

Getting your email opened is only a job half done. After all, your email campaign’s end goal is to motivate the subscribers to take action, which requires them to read through the copy.

Now, several aspects affect the amount of attention the email recipients will dedicate to it.

Some of the essential elements of the email body are:

1. Tone and voice.
   There are a lot of different ways in which you can convey the same piece of information. Altering the tone and voice that you use when delivering the message can change the message itself. It’s all about the things you accentuate on.
   
   Upbeat narration helps the reader understand your key message and encourages them to click-through and purchase your offer.
   This, however, varies greatly depending on the type of the email campaign you send and the audience you target.
2. Length.
   The more, the merrier? Could be. But apparently not in email marketing.  
   Ever since the smartphones and the internet, people’s attention span has decreased to under 8 seconds, which is a second less than that of a goldfish. While it might not bother you in everyday life, it’s definitely something you should consider as a marketer.
   
   Modern people don’t have the time to read through long emails that fiddle around the point without actually getting to it. You want your email to be as concise as possible, keeping the word count to the recommended 50 to 125 words.
3. Visuals.
   You have to be careful with visuals, too. While it’s a great idea to incorporate a couple of images into your copy to spice the email up a bit, you need to remember that they are not always displayed correctly.
   
   Besides, images are one of the most controversial parts of email marketing as they’re so subjective. You can’t just assume that your subscribers’ taste coincides with yours.

3. CTA

Nailing your email’s CTA is a must if you want to encourage your subscribers to take action upon reading your email. Otherwise, it’s a waste of money.

To make sure your subscribers take the action you expect them to take (either that’d be joining a community, downloading an e-book, watching a video or purchasing a product), you need to show them in the right direction.

The only way in which you can do that is by including a Call-to-Action into your email copy. A well-designed CTA is your best friend! But there’s a lot of controversy regarding what makes a good CTA:

• Shape
• Button or Text
• Length
• Colours
• Positioning

We’ve talked plenty about CTAs and A/B testing in one of our previous posts. Make sure you check it out to learn everything you need to know about making a bomb CTA.

Then, you can A/B test common best practices on your mailing list subscribers to find the one that generates the most conversions. Even the slightest change can cause a butterfly effect.

4. The ‘From’ Line

This one is less obvious than the rest on the list. Nonetheless, it’s still crucial to optimise. Especially since with the tightening of the privacy laws in Europe, including a dodgy sender name can result in a fine.  

Perhaps, your emails will get more recognition if the sender name is more human-like as opposed to being official. For example, you could test ‘NetHunt’ against ‘Valerie from NetHunt’ to see which one performs better.

5. Send Times

Last but not least, you need to find the most optimal time to send out your emails. Even if the rest of your email campaign is perfect, sending it at the wrong hour can ruin all your efforts and result in an unsuccessful campaign.

Below is the results of a study conducted by Mailchimp:

Send times, however, are the most personal out of all email campaign elements that need to be A/B tested. There are numerous factors that affect the best send times and are unique to an individual company and its subscribers.

Email A/B Testing Tips and Tricks

Once you know what you can test, it’s time to learn how to do it. We’ve already talked about major A/B testing practices, so make sure you check out those before moving on to email marketing-specific ones.

Don’t focus on too many things

Knowing about the benefits of email A/B testing, it can be tempting to test all of them at once. But this is exactly what you shouldn’t do. Not only is it physically impossible to run numerous tests simultaneously, but it’s also extremely expensive to do so.

Instead, you should pick the most important elements of your email campaign to test first, and then move on to the less crucial ones.

Use reliable software

A/B testing is a very quantitative method of improving your email marketing performance. It requires accuracy and attention to detail. Therefore, you need to make sure that you accommodate that through using high-quality A/B testing software for your email campaigns.

Some of the best email A/B testing tools are:

• Campaign Monitor
• Mailchimp
• Send Pulse

Decide on the size of your test

Usually, you’d want to test your entire mailing list. That way you have a larger test sample and, therefore, can get more accurate results. However, there are several instances when you shouldn’t go for every address on your list and opt for a smaller sample instead:

• Very large list + high A/B testing cost. If your A/B testing service provider charges per address and you have a lot of contacts on your mailing list, you might want to limit the sample to the largest you can afford. Just make sure that the contacts you do include into the A/B test are representative and are chosen to accommodate for the most accurate results.
• An extreme variant. If you’re trying something risky, you might want to limit the number of people who see it in case it doesn’t work the way you intend it to.
• A limited-time offer. If you’re tight on time, you might want to run a small (a few hundred recipients) test batch first, and then send out the winner to your entire list.

A/B test your emails to be able to predict what effect your email butterfly will cause.

Table of Contents

Originally published February 22, 2021, updated June 05, 2023