Smarter split testing

The goal of split testing is to maximize conversions on your website.  A conversion can be any desired action or change from your users.  The general reason for using a split test is to be able to accurately measure the affect of your change on your conversion rate.  Assuming the change is in your favor, you can safely implement the change for all of your visitors without having to worry about the error of assuming your visitors will like your change.

The biggest problem with split testing is that it doesn't always increase your conversion rate.  In fact, there are three possible outcomes of a test.

• Increase conversions (by up to 50% of the possible increase)
• No change to conversions
• Decrease conversions (by up to 50% of the possible decrease)

The followup issue is that if you do come up with a positive change, you won't be able to fully realize the affects of your change until you stop the test.  

For high traffic websites, this may not be an issue, but for a 5-10% increase in conversion (depending on your current conversion rate), you need 100,000 - 1,000,000 visitors to reach statistical significance.  Meaning the results of your test are not random.  If your website gets 1000 visitors per day (nothing to sneeze at), it would take at least 100 days before you can comfortably stop the test.  

If your change would win you a 10% increase in conversion, 100 days of only a 5% increase (the amount you can realize during the test), isn't too bad.  But, if your change is a 10% decrease, that means for the next 3 months, you will have 5% fewer conversions than if you didn't test at all.

This issue is pervasive in the A/B testing world, and I have decided to do something about it.  I am starting a project at GreedyTest.com to create a split testing tool based on a modified Epsilon-Greedy algorithm that will maximize conversions during the testing period.  

The GreedyTest algorithm dramatically changes the possible outcomes of a split test.

• Increase conversions (by 98% of the possible increase)
• No change to conversions
• Decrease conversions (by only 2% of the possible decrease)

Note: the worse case scenario for a test is a decrease of only 2% of the potential decrease.  If you have a 2.0% conversion rate and the "bad" option has a 1.8% conversion rate, an A/B test will have a conversion rate of 1.90% during the test.  The GreedyTest will have a 1.996% conversion rate during the test.

The project is just getting started, but if you want to stay up to date with our progress and be the first to know when we are ready to launch, go to GreedyTest.com and give us your email.  A smarter testing platform is on the way.