BIG DATA is a loosely defined term to describe data sets so large and complex that they become awkward and overwhelming to work with using on-hand database management tools like Excel and Access.

Wal-Mart, a big data example, feeds more than one million sales transactions an hour into a BIG DATA set with a capacity of one million gigabytes (2.5 petabytes). Try putting even a couple of hours of this data in an Excel spreadsheet!

BIG DATA is big and getting bigger. By all accounts, we are in the early years of the era of BIG DATA.

The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same data. The sum is more valuable than the parts.

In the semiconductor industry where blank silicon wafers are converted to individual computer chips, the data set of an operations reporting system like work stream has thousands and thousands of columns and millions and millions of rows of data. Selecting a subset of this data for a spreadsheet analysis may not take advantage of the larger data set. The sum is again more valuable than the parts.

There are significant differences in the way industries and people use and require data, summarized into four broad groups:

Data Wasters. They have lots of data but never use it. Sound familiar?

Data Collectors/Hoarders. Submerged in data, they collect everything and don’t do much more than store it. “In Health care, it’s not ‘big data’, it’s a title wave of data” says a Corporate VP of Quality and Medical Management at a big hospital in California.

Aspiring Data Users. Fully embrace the importance of BIG DATA but lag behind the leaders. Retail and communication industries are in this group.

Strategic Data Use. Use and make decisions on a large percentage of the data they collect. Manufacturing, financial services, and technology companies are in this group.

The challenge for each of us in the performance management community is to master new tools and methods to extract value and usefulness from BIG DATA sets that result in better business models, better informed decisions, better profitability analysis, better performance measures, better product/service costing, better productivity, better business plans, better projections of the future, better quality, and better customer satisfaction.

Get to Know your Data Source…

John A. Miller

Leave a Reply