Monday, February 18, 2013

Accurate or Usable?

Many great minds including William of Ockham (Occam) and Albert Einstein have come to the conclusion that solutions should be as simple as possible.  While "as simple as possible" implies that over simplification isn't any better than over complication, there seems to be a void between accurate and usable solutions.  

In statistics, higher levels of accuracy are achieved by adding complexity to the system.  If you could take into account enough variables, you could rather accurately predict the future.  But complex systems are highly vulnerable to errors, generally have a steep learning curve, and are hard to use and maintain.

Our mind solves the issue of over complication by looking for indicators that something familiar is about to happen and then turns on "auto pilot" to navigate us through pre-charted waters.  In business, we often try to do the same by looking for easy indicators vs worrying about the actual, complex results (which may come too late to be relevant or be too difficult to measure).

Google Analytics, along with almost all other third party analytics software, is notoriously in-accurate.  They gloss over the fact that all the numbers are wrong by highlighting their use for analyzing trends.  

A/B testing software is based solely on conversion rates, and disregards all other business indicators that would suggest a meaningful increase in business.  They brush off this criticism because conversion rates are generally a strong indicator for business.

To have a truly accurate analytics report, you would have to collect and maintain the data yourself.  To run A/B tests that take into account your other important business indicators, you would have to have a custom solution that hooks into your analytics and business data.  Creating and maintaining this level of accuracy would be difficult and require vast resources, not to mention how easy a tiny mistake could bring the entire system to its knees (or worse, incite the wrong business decision).  Systems like this don't scale to the outside world.

It seems the question in the modern digital world, isn't "how simple can we make it?" but rather, "How accurate can it be and still be useful?"