Saturday, November 3, 2012

The significance of significant

What does it mean to have statistically significant results?  Are insignificant results irrelevant?

To answer these questions, we have to look at what it means to have statistically significant results.  If you have two sets of results, A and B.  
A: 1624 visitors; 72 conversions; 4.43% conversion rate.  
B: 1609 visitors; 96 conversions; 5.97% conversion rate.

B clearly has a higher conversion rate.  Statistical significance helps tell you how likely it is that B will ALWAYS have the higher conversion rate.  

Statistical significance makes an estimate of what the best possible conversion rate will look like and the worst possible conversion rate.  If the worst day for B is still better than the average day for A, then you have a statistically significant result.  However, if the worst day for B is lower than the average day for A, than it is probable, A will actually have a better conversion rate than B on any given day.

In the example above, B's worst day is still better than A's average day, so the results are significant.  In this case we have a 98% confidence level.

But what if the confidence level is low?

Statistical significance is only one tool to help you determine whether a change is good or not.  It is important that you also look at volume, time, revenue, and trends.

If the difference in the volume of conversions is significant to your business, then the results are significant.  In our example above, the difference is 24 conversions.  If 24 conversions, during the time frame of the test, is big for the company, then the results of the changes should not be ignored.

Time is another important factor.  Too little time means you don't have enough natural variation and randomness to demonstrate confidence in your numbers.  If you reach statistical significance in 1 hour, its hard to imagine it will hold at the end of the day.  Results like this often mean you should segment your traffic vs make broad, sweeping changes.  

Revenue can also be a game changer.  If A above generated more revenue because it increased cart size, the conversion rate suddenly becomes less important.  I would suggest testing a hybrid of A and B to both increase conversions and cart size.  

The last thing you want to pay attention to are the trends.  If B is consistently above A on a daily basis, then you have nothing to worry about.  However, if B has a high variability, meaning some days it is very low and some days it is very high, you may want to reconsider.  It can be hard on a company to have an ebb and flow of orders.

Statistical significance is just one part of the story.  It is important to take an overall view of your changes to make sure you are making the right choice.