Why Big Data is a Big Deal

Most companies have noticed that "Big Data" is everywhere.  It seems everyone is talking (and writing) about big data.  So what is it?

Big data is just a term to refer to the largest data sets available today.  Big data is often referred to in an online context because tracking software can capture a ridiculous amount of data from any web user, not to mention the amount of information a user inputs and consumes while on the web.  All this information creates a rather large and unruly data set: Big Data.

The biggest challenge of big data, is being able to crunch it into usable pieces.  We have the technology to capture this data, we have the hard-drives to store the data, and the tools to look at what data is there, but we don't have a way to truly understand what it all means.

If we don't have an easy way to understand all this information, then why is it such a big deal?  

The data that we process has always progressed at the rate that we are able to collect and store it.  In the late eighties, most servers could only collect and store a few hundred megabytes worth of data.  This level of data was relatively easy for us to process.  In fact, most processing was done by hand because we humans could put dots on a map faster and more efficiently than a computer.  

Today, we can collect and store so much data that we don't know how we should process it.  Big data implies data so big, we can't do much with it.  Yes, we can get some insights from breaking down various sections of this data, but a true comprehensive analysis would take far too much time to be useful.  

Big data is a big deal because processing ability is now the limiting factor and our ability to collect and store new additional data isn't slowing down.

Do you need to be concerned about Big Data?

Large companies are using big data to dig deeper into customer habits and patterns.  This allows them to target their marketing efforts to the right people at the right time.  It also lets them predict which stores should stock up on given inventory.  Essentially, it lets them make smarter business decisions.  These decisions are also easier to sell to the stake holders because they are based on verifiable data, not intuition.  

If you are a small business and your intuition continues to push you forward, the cutting edge of Big Data Analysis may give your business an unnecessary hair cut.  If your business is flailing in this new digital world, it may be time to see what some data analysis can do for you.  

How does your company make decisions?