7 Myths about split tests

Split tests or A/B tests are the best way to determine the impact of your changes.  They are fairly simple to perform and more people could benefit from doing them more regularly, but there are a few myths that I believe are holding people back.

1. You will get results - Not all tests are created equal.  Actually, most tests don't have a significant impact on the business (maybe why more people don't test.)  If you do your homework and get insights into what affects your visitors, you have a better chance of crafting a significant test.
2. Test everything - "Everything" doesn't matter to your visitors.  I don't buy a product online just because the "Buy Now" button is red.  On the other hand, if I can't find the buy now button, there is a good chance I will go somewhere else.
3. Statistical significance is all that matters - Statistical significance is only important in the world of statistics.  Statistical significance is a good indicator that your test is valid, but make sure that your results are good for the company as well.
4. Your results will hold - Not all results stand the test of time.  This happens for a number of different reasons, but suffice it to say, the world, your visitors, and the web is constantly changing.  Just because people are worried about security today, doesn't mean they will be tomorrow.
5. Never show your test to returning visitors - If you have long time visitors to your website, on one hand, you don't want to shock them with a crazy test, but if your test is valid, a little improvement may be all they need to convert.
6. Multivariate testing (MVT) is the holy grail - MVT rarely works.  If you want to find out which one of 3 ads works best in which one of 2 spots in combination with one of 3 headlines, you are looking at 18 different combinations.  If a simple A/B test takes 2 weeks to get enough visitors to be relevant, you could be looking at 18-36 weeks before you get a positive result from your MVT.  If you decided to test all 18 combinations with A/B tests, you could realize incremental results every 2 weeks.
7. You should always be testing - Not all websites get enough traffic to do split testing efficiently.  For those that do get tons of traffic, if you have a high number of returning visitors, too much change can make the site feel unstable.  Every situation is different, but you don't want to loose confidence from your returning visitors by constantly changing the core of your website.

If you're thinking about doing a split test or are already doing them, hopefully this list will help you avoid some of the pitfalls.  If you have found other myths or unexpected results in your split tests, please share in the comments.