Slow and Steady A/B Testing

Slow and Steady A/B Testing

Nov 27, 2017 by Myca A.

View Myca's profile

Digital marketing is an art as well as a science. Discovering how to convert more leads on your website may feel like a modern-day form of alchemy to the uninitiated. If you're asking yourself, "what exactly is A/B marketing?" and "why are people so rabid about doing these tests for every campaign?" we have the answers.

Creating a standardized process for A/B testing important campaigns and analyzing the results provides you with ongoing knowledge of your customers and helps you see first-hand how they are likely to respond to future offers—important information for marketing teams when developing strategy and tactics. Here's how to separate the important variables from the less-important ones, limit confusion, and still optimize campaigns to boost results.

The Basics of A/B Testing

Have you ever struggled with writing an email subject line and wondered why one particular message resonated more with your audience than others? If you're fortunate, you have a methodology in place to test various directions before launching a full campaign. Luckily, even basic email software packages provide you with A/B testing capabilities.

Essentially, A/B testing (also called split testing) offers a way for marketers to experimentally compare two versions of the same piece of content to determine which one is more compelling to their audience based on statistical evidence of engagement. You'll often see A/B testing with these content types:

Email subject lines
Email preheaders
Email body text or images
Landing page color schemes
Button colors on landing pages, in emails, and in apps

Testing these individual elements in a granular way provides marketers with direct feedback from the audience—votes, if you will—that show their reaction to controlled content variations. When used consistently, testing one change at a time provides feedback that helps refine the message and drive additional engagement, depending on your selected conversion metric.

Determining Conversion Metrics

Which metrics you choose to measure depend in large part upon the campaign and the test options. For instance, in an email campaign, the most important metric may be clicks-to-open, or it may be conversions based upon the email. Marketers may also decide that a blending of these two metrics provides the best results, as optimizing clicks can be done within the email, but the conversion actually happens on a landing page.

What is important is defining success measures and testing points, and then clearly stating them as part of your overall marketing campaign strategy. After all, you can’t have a successful campaign if you haven’t defined what success is.

Conversion Rate Optimization

Ultimately, all marketers are looking for the mythical Conversion Rate Optimization (CRO). How marketers define a conversion can have a wide range of answers, such as signing up for a newsletter, downloading a whitepaper, or completing a purchase either on your site or in a store.

If you’ve researched CRO, you may have realized that there are tons of blog posts out there attempting to convince readers that there are definite tactics you absolutely must be doing. It can be completely overwhelming to try and implement all of their suggestions. Fortunately, optimizing your conversion rate doesn’t have to involve a laundry list of someone else’s best practices. It can be as simple as implementing structured A/B testing at each stage of your sales funnel. Listen to your own audience and you’ll find the best possible conversion rates.

The KISS Principle

It can be tempting to define all the possible variables within a campaign and start testing them all, but you’ve got to resist! The best tests are those that use the KISS Principle: Keep It Super Simple.
If you're testing more than one variable at a time, then there's too much room for confusion when you attempt to analyze test results. When you have a clear baseline and only test one variable at a time, it's easier to determine which move worked.

Testing with Confidence

If you're sending an email to thousands of people as part of a test, resist the urge to stalk the results page. Waiting to see which variable wins before hitting a button to send the campaign to your full database isn’t the best way to go. Instead, ensure that your results reach statistical significance before you declare a winner.

There are a number of factors that can influence a test, such as the time of day or quickness of your audience to check their email. If the accuracy of your test is quite low, you may want to consider additional variables before declaring a winner. Similarly, when testing a subject line, you’ll want to consider the click-to-open rate and the total clicks in addition to just the open rate as part of the overall analysis.

Testing procedures are an incredibly important part of the science behind digital marketing. If you take the time to test in small increments, you’ll be well on your way to optimizing the variables.

made by Freepik from

Enroll in Email 101!

Back to blog