Friday, September 21, 2012

Statistically Significant

If you're running a split test manually, it can be hard to know when the results are statistically significant.  Some online calculators give you p-values, z-numbers and confidence intervals, none of which mean much without a stats book and a graphing calculator.  Others give you a simple yes/no answer.  To me, the simple yes/no is easy, but it doesn't really tell the story of the data.  

This morning I set out to "fix" this situation.  I wanted a visual representation of the data that was easy on the eyes and brain.  So, after a conjuring up old memories of sleeping in stats class, I came up with the idea of using venn diagrams to represent a 99% confidence level in two samples.  See my visual Confidence Calculator.

If the circles overlap the opposite label, the jury is still out.
If the circles overlap but don't overlap each other's labels, its time to keep the winner and do a new test.
If the circles don't even touch, you've already waited too long to call the winner.