Friday, November 23, 2012

Greedy algorithms for holiday split testing

If you're looking to test different offers or site elements this holiday season, consider using an epsilon greedy algorithm instead of a simple A/B split test.

A/B split tests direct 50% of your traffic to A and the other 50% to B.  This is the quickest way to achieve statistical significance.  But if one variation, A or B, is significantly better than the other, you sacrifice valuable conversions because a split test will continue to send traffic to the poor performing option.

An epsilon greedy algorithm, used to solve the multi-armed bandit problem, maximizes your average conversion rate during the test.  It keeps track of the current conversion rate of both of your options and sends the majority of your traffic to the one with the best conversion rate.  A small percentage of your traffic is used to continuously test the lower performing options.  You don't reach statistical significance as soon, but your average conversion rate during the test is higher.

If you're planning on continuously testing your site this holiday season, don't just get answers, maximize your conversion rate at the same time with an epsilon greedy algorithm.

Visual Website Optimizer's explanation of split testing vs MAB algorithms.

A simple greedy algorithm.