Why multivariate testing isn’t always a good thing
I’ve been thinking about multivariate testing lately, and have decided to collate my thoughts on why it’s no silver bullet, comes with many risks and should be used only in the right circumstances.
Why test at all?
If you have a site resulting in a call to action there are many things you can alter that might increase take-up.
If you just change things without testing you run the risk of making things worse, or lack a true understanding of what helped to create an uptake.
Create a hypothesis, a series of design suggestions, and test them on real users by using multivariate or split testing.
What is the alternative to multivariate testing? (its split testing)
Split testing allows you to test users on completely different versions of the page, where you could change one or more items, or run an entire redesign. You can test one, two or more variations of the page. A/B testing is term used to describe split testing of 2 alternatives.
Once you have completed the testing, you have a winner and you can make it live.
What is multivariate testing?
Multivariate testing allows you to change a number of elements on the page, and run tests to see which of those elements are more likely to result in improvements to the call to action.
Visitors to the site will see different element combinations each time, and the system you use will attempt to calculate which items are proving more successful over time.
Once you have completed the testing, you need to create a new design based on your learning.
The primary benefit of multivariate testing over split testing
Multivariate testing has a lot of fans, large companies have seen increases in profits thanks to its use. You will find many avid proponents of its use, especially from those who own companies that sell it as a service.
The primary benefit of multivariate testing over split testing is you can try a number of changes all at once, run the tests, and find out approximately which were most successful, growing your knowledge of what triggers your customers to complete a task. If you are going to put this knowledge to good use in the future, then it is good to know. This is one of the principles of the lean startup movement.
To achieve this in a split test, you would need to change one thing at a time, see if it worked, and then if not, roll back and start again, so *it would take longer to gain that knowledge.
However, you could also make a lot of changes in a split test, see your profits increase
The downsides of multivariate testing
It is complex to set the tests up, whatever the blurb may say
The tools such as market leaders Visual Website Optimizer and Optimizely both work by allowing you to edit the HTML and CSS inside their apps. Then they roll the changes out onto your site using JavaScript.
Setting up the changes is fiddly and requires some HTML and CSS knowledge anyway. It should be easier for a skilled front-end developer to set up experiments in a split testing environment where you just upload an alternative file. If you aren’t a skilled developer you probably shouldn’t be making the changes anyway.
You need to plan very carefully
You need to think about how all the various combinations of changed elements could look together. You don’t want your site looking like a mess, as that will put people off anyway and skew the data.
It could harm your search engine rankings
Any split or multivariate testing can impact negatively on your search engine rankings as you run the risk of being penalised for duplicate content, lost backlink equity and page slowdown. This SEO company explains it all in more detail.
Whats concerning is that whilst there is plenty you can do if you are running split tests to mitigate the risk, with multivariate you are powerless to do anything. You just have to assume the risk, and take the hit if it occurs. For this reason alone,** I wouldn’t use multivariate testing on a site that wasn’t extremely well linked to and established.**
You can easily destroy cross browser and multiple platform consistency
When setting up the multivariate tests you have no way of easily testing the changes across multiple browsers and platforms. It certainly needs to be done, but it will take you much longer to do when running multivariate tests than split tests. In a split test you can simply load the 2 pages, and test them manually. With multivariate tests you will need to hope your chosen software makes it easy to force certain scenarios and then work through them all.
Multivariate testing also gives you the potential power to destroy your site, so make sure you trust the HTML and CSS knowledge of whoever has control of the tests.
Its takes a long time to find a potentially inaccurate answer
Multivariate testing doesn’t ever truly prove that a certain element was responsible for the conversion rate increasing. It can only strongly suggest it. Since every user sees a different combination of elements, you can’t be 100% sure it was the element in question which convinced them.
To be even mostly sure as Paras points out you need to see a large increase in conversion rates when a particular element is present, have a large number of visitors and ideally be starting with a page with an already high conversion rate of around 50%-60%. This takes a long time, according to Optimizely ‘Even a site with fairly high traffic might have trouble completing a test with more than 25 combinations in a feasible amount of time.
You still need to do a design at the end of the tests
Multivariate testing will produce some pointers, but then you need to create a new design with the results taken into account. There will be more coding to be done, and you might not be happy with the hodgepodge design that will be produced. Plus, that particular combination of elements might not even work together…so…
You still ought to do A/B testing of the new design vs the old
Otherwise, you aren’t testing the final release - the most important of all. Expect to wait around 2 - 4 weeks for that to be completed.
The software isn’t cheap
Expect to pay between $100 and $400 per month to run multivariate tests.
If you trust your redesign ideas, or have an under performing site you could profit quicker using split** testing**
If your site has obvious issues, that you can see based on intuition and studying more successful competitors, such as:
- an outdated design
- contains quickly drafted copy, that you wrote without outside help
- has not been built on solid design principles and sales ideas
Then **it is likely that changing these things will result in higher sales. **
You don’t need to know which of your individual improvements lead to the higher sales straight away, you just need a stronger bedrock to build on.
Why spend lots of time and money running multivariate tests when you can clearly see the improvements are there to be made.
Make them, ensure through A/B testing that you won’t actually see a drop in profits, and get there faster.
If you doubt your redesign ideas, or have no clear redesign in mind, you can increase profits slowly over time with multivariate testing
If any of the following apply:
- you can’t see immediately where you would improve the site
- your site is built on solid design principles
- you and others perceive your site is as good or better than your competitors
- you have range of ideas you might implement but are unsure of
- you have strong current conversion rates
Then multivariate testing could be for you, especially if you have a solid knowledge of CSS and HTML.
Try some ideas, and see if any of them make a difference. If they do, update the site, if not, it’s no problem.
You won’t save a failing business this way - it’s too long-winded, but you might improve upon the already successful.
However if you don’t have an abundance of time, visitors or money, beware of multivariate testing
This sums it up for me:
…A/B testing is so speedy and easy to interpret that some large sites use it as their primary testing method, running cycles of tests one after another rather than more complex multivariate tests.
Optimizely - sellers of multivariate testing software
Conclusion
You need to choose the right tool for the job. Each are valuable:
Multivariate testing requires more work, takes longer, carries more risk, but can provide arguably more accurate results.
A/B testing is simpler, quicker, less risky but you won’t always know whats worked unless you test one thing at a time.
comments powered by Disqus