Testing is central to marketing and SEO. I have researched and worked out how multivariate testing works. It is much better than the standard “A/B Split Test” as you can test multiple elements at the same time, making the testing process a LOT quicker. Multivariate testing has many different forms, depending on what you are trying to achieve. Taguchi testing is a subset of multivariate and is generally recognised to be the best place to start your calculations.
If you want to do a multivariate test, you either have to buy software, such as Kaizen Track or Split Test Accelerator (no affiliation) or wade through hundreds of pages of waffle on the search engines to find what you are after.
Well I’m sick of that. I want to be able to do tests and understand the process… for free. So I have done the work and hopefully someone will find this useful and not have to go through what I just did.
To set up your test, you need to work out how many elements (parameters) you are going to test and how many variations (levels) you are going to have of each element.
To do this type of test, you are going to use an orthogonal array. This is a statistical way of working out which combination of elements you need for each test, along with how many tests you need, so that you can be statistically confident that you have isolated the results for each element and can see which variation of each element will give you the best results.
Download this file – multivariatecalculation.xls – which I wrote to explain how the calculations work.
Go here and work out which orthogonal array you need for your tests. Then follow the appropriate link, which will take you to a picture of what combination you need of each element for your tests (as well as how many tests you need to run). For example, say you need to test 3 elements (good for adwords or for testing heading, sub-heading and price), you would need to use the L4 array.
The excel calculator i wrote only applies to the L4 array, though it is easy to change the calculations for another array. Just play with the calculations until you understand how it all works.
Run your test, put your results into the calculator and you will be able to see what, statistically, the best variables for each element are. Run this new combination against your original control (ie the page or ad you were using before you started this test) to see whether you get a better result (whether conversion, optins, etc). If you have run the test for long enough (the more variables and elements you use, the longer your test needs to go for before you can be certain of the results), there is a very good chance your new ad or page will outperform your original one. How long is long enough? It depends… you just have to make that call yourself, but it is better to err on the side of running the test for too long, than not long enough. The longer you run the test, the more certain you can be of the result.
Keep testing. If you have any queries or you want something explained further, just leave a comment to this post and I’ll get back to you.
More from the blogosphere…
- Speeding Up Taguchi??
Leave a Reply