For geeks like me, multivariate testing is some of the coolest stuff in the world!
A/B Tests are extremely powerful, but you have to be careful to measure the impact of how several variables work together to affect your conversion rate. In the majority of cases, combining the “winning” A/B tests will improve conversion rate, but this is not always guaranteed.
In this post, I’ll give a quick summary of multivariate testing along with some examples of how to use it to improve your website’s conversions.
What is Multivariate Testing?
Multivariate testing is a process for measuring the impact of multiple variables on the conversion rate of your website.
For example, let’s say that you’re looking to do a multivariate test on a website such as shoes.com:
Like any ecommerce site, there are tons of ways of variables to test in a single A/B test:
- What offer performs best on the homepage?
- What copy is the strongest?
- What should the navigation look like?
- What’s the ideal call to action on the sign up button?
But, the real interesting part comes when you start measuring the impact of variables combined. For example:
- What’s the ideal background color and homepage headline?
- What’s the ideal combination of the homepage headline and the email call to action?
Etc – the typical e-commerce site has 20-40 different variables that can be tested! With so many different elements, it’s important that you’re able to accurately measure the impact of each variable both individually and in combination with another.
Examples to Improve your Site
1. Landing Page Headline & Copy
Obama Campaign
One of the best (and simplest) examples of a multivariate test is the following combination of background image and CTA from the Obama campaign:
There are four variations of this landing page to be tested:
- Get Involved Image with Sign Up CTA
- Get Involved Image with Learn More TA
- Change Image with Sign Up CTA
- Change Image with Learn More CTA
(The full test had more variations, but we’ll use this for simplicity)
The result? The “B” version generated 40.6% more email leads for the campaign, which roughly translated into $60M in additional revenue.
You can replicate this testing pretty easily on any of your landing pages with call to actions or images. For example, see HawkHost’s multivariate test of landing page copy and ad background. Here’s the original creative:
HawkHost identified two key variables to test:
- The headline – “Hosting You Can Trust”
- The image used (in this case, a globe)
They then ran a multivariate test using:
- Three different headlines
- Three different images
That made for a total of 3 x 3 = 9 total tests. The following combination was the winner:
What’s most interesting about this multivariate test is that it looks, at first pace, like a simple A/B test with a new image. But in reality, none of the other headlines performed as well as the original – it was the new image that really boosted response.
In general, for any landing page, you can combine variables and multiply them to determine the total number of multivariate tests. For example:
- 2 new headlines
- 2 new images
- 4 total new variations
It’s important to understand that testing multiple variables at the same time will increase the amount of time and traffic that you need to reach statistical significance. But, for sites with large traffic where a conversion rate boost translates into more revenue, this can be well worth it.
2. Free Shipping & Placement
Here’s another example for multivariate testing – how do you determine the best offer for shipping? What’s the optimal price point for free shipping? And where do you place the offer for conversion?
We can use Edible Bloom’s current website as an example:
The current site offers “Shipping NZ Wide”. Even with this simple offer here, there’s a number of things you can test:
Placement
Should Shipping Across NZ be placed in larger font? Should other offers be removed in order to make it clear that this is a benefit?
Price Point
Should shipping be made free, and if so, at what price point?
Shipping Time
Should you offer next-day shipping, or 2 day shipping? How soon do customers want the product?
While these sound like simple tests, I cannot state the importance of optimizing your shippin Getting the perfect offer for shipping can lift e-commerce conversion rates by as much as 25% – 40% when executed properly.
In a good multivariate test, you would layout the variables and combine them. For example:
- More Prominent Shipping + “Delivery NZ Wide”
- More Prominent Shipping + “Free Shipping”
- Less Prominent Shipping + “Delivery NZ Wide” (control)
- Less Prominent Shipping + “Free Shipping”
Because there are four versions of the site to be tested, this test will take additional time, but you will know at a very high level of confidence what the optimal offer is.
3. CTAs & Pricing
In a software company, multivariate testing can be used to combine the impact of CTAs on the homepage with the optimal price point. For example, take UserVoice’s pricing page:
There are *tons* of variables to be tested here, and they likely have enough unique visitors to justify extensive testing. But, at a basic level, you could look to test two variables:
- “Sign Up” CTA vs “Start for Free”
- Removing the Enhanced Plan
Then, your multivariate test would look like this:
- Sign Up with Enhanced Plan
- Sign Up without Enhanced Plan
- Start for Free with Enhanced Plan
- Start for Free without Enhanced Plan
A test like this would allow uservoice to determine both what call to action works best as well as start to understand what pricing plan works (although that testing would be enough for another article).
Summary
Multivariate testing is simple and will let you quickly determine the optimal combinations to maximize conversion. You can easily get started today by testing headlines, images, CTAs, pricing, and placement.
What do you plan on testing?
Author Bio
Andy is the founder of Uplift and is responsible for all optimization competitions. You can follow him on Twitter at @upliftroi.
