Our blog

This week, we have a guest post from Whereoware Web Marketing Manager, Peter Bergen. Recently, Peter’s A/B test for Paragon won Honorable Mention in the B2B Email Blast Templates category during Which Test Won’s 2012 Email Test Awards. This test was also added to the 2012 Email Tests Hall of Fame. Here, he breaks down his experiment, and shares secrets to making A/B tests work for you.

A/B Test: Paragon

Purpose: The goal of this test was to see which approach to a new product introduction is better – a “creative” approach or a “standard” approach.

What does this mean?

Two different designs were prepared:

• Standard design
  • Straightforward header and overall message (“Introducing over 200 new products”)
  • Button-supplemented header text (“See all new products >>”)
  • Simple paragraph of text with bullet points describing new products
  • Used html text (shows up even if images are disabled)
  • Many neatly-arranged product images
     
• Creative design
  • Header text and overall message tried to describe one of the themes behind the new designs (“Art doesn’t have to match the couch”)
  • Call-to-action (i.e. “Come see our new products”) was embedded in button rather than in header text
  • No explanatory text
  • No html text
  • Not many product images

Sometimes, it’s okay to be the same

These two versions weren’t entirely different, however. In order to make an accurate comparison, the emails shared the following controls:

• Same subject line (“Introducing over 200 new products”)
• Sent on same day at exactly the same time
• Placement of the call-to-action at top of email

And the winner is…

In the end, the Standard approach was the clear winner, pulling in better results across the board, including Open Rate, Click Rate, Revenue, and Number of Transactions:

Main takeaway

One of the main lessons learned is that the header text must be clear and straightforward so that people immediately know what the email is about. The call-to-action/button can and should supplement the header text, but it can’t replace it.

In addition, as with any A/B test, there are caveats:

• The designs were very different and the test did not isolate any variables
• While the standard approach was the clear winner, it’s also possible that we didn’t take the right “creative” approach – there may be other creative ways of presenting Paragon products that we haven’t come across yet.

So what?

You’ve heard it before, you’ll hear it again: test, test, test! In this case, Paragon saw a clear benefit (a 20% higher open rate, nearly 5% higher conversion rate, and almost 5 times as much revenue!) when using one email template rather than the other. And testing doesn’t end here – the smart marketer will take these results and set up the next campaign with the same criteria in mind, keeping the header text clear and straightforward (now shown to be a successful strategy) while tweaking another element. The results of that test will then inform further tests, on and on and on, allowing a company to continually optimize the results.

The bottom line: you can go with what you ‘think’ is right…or you can take advantage of testing + let customers tell you what speaks to them. After all, they’re the ones bringing you revenue, so you’d be wise to listen to what they’re trying to say!

Like what you see?  Check us out on Facebook, twitter, or Pinterest!