Thursday, September 27, 2012

Too small to split test

Most Conversion Rate Optimization Consultants view split testing as the Holy Grail.  Split testing is when you direct a portion of your traffic to your control, and another portion to one or more variations of the same page.  Split testing is important because the variation that works the best isn't always the most logical or appealing.  If your tests show statistically significant results, you can't go wrong with split testing.

But what if you don't have enough traffic to run a split test in an efficient manner?  If you only have a few hundred visitors each month, a split test where one variation shows twice as much success could take up to 3 months to prove statistically significant.  

The answer is:  There are other ways to measure the success of a variation.  

  1. Usability Tests:  If you run a usability test before the variation and another one after, you can compare the results and see if you have fixed the problem.  
  2. Surveys:  You can also do a short survey before the change and ask if participants would be willing to let you follow up with them.  After you've made your changes, follow up with previous participants and run the same survey again for new visitors.
  3. User Panels:  I suggest forming 2 user panels.  The first one to consist of employees of your company and/or persons close to the company and the second of people who make up your "average" visitor but don't have any relation to the company.  This lets you see your website from both sides of the table.  Asking for a review from each panel before and after the change can help you determine if you've achieved your goal.
Split testing can be the easiest way to measure the results of your changes, but if you don't have a lot of traffic and still want to improve your site, give these other methods a try.