Is BIG Data such a BIG Deal?

[yes it is about technology, but it is also about managing]

by Richard Smith.

You may be familiar with the cries of the prospectors in days of the Gold Rush: “There be gold in them thar hills!” Well, you would be forgiven if you saw similarities with all the activity around Big Data these days.

It is like the American Wild West and the early days of mining in other ways, as well. There is a vast area that is as yet unmined; there is a rush to claim a stake; some of the people involved are miners, but more are people who have never mined in their lives but are just caught up in the rush; the tools are the equivalent of pick axes and shovels (although some heavy machinery is moving in); everyone is struggling to put a value on “the find”, and starting to decide what to do with the waste (in the longterm you cannot afford to have piles of Big Data scattered around your organization).

Who made money during the Gold Rush? Yes, some of the miners did strike it rich, but the vast majority did not. Did the jewelers make money, did the grocery store owners make money? Did the outlaws who found it easy to get their hands on the “data” because control and law and order did not catch up with the rush on time? Did the infrastructure people, the railroads, make money? In the early days of the Gold Rush, no one knew what would happen and there were piles of dirt and broke people scattered across the wilderness before it all settled down.

I believe that “There be value in them thar piles of data,” so it is worth your time to look for it. Every now and then someone will strike gold and will find a new mineral such as DataInformium*, which will be valuable in and of itself. But the rest of the mortals will find valuable nuggets because the process of searching will have forced them to improve the products or services that they have been offering. The secret to the value of mining big data will be taking some action to improve what you have been doing in your businesses, tweaking it, focusing better, or even eliminating the wastefulness, based on what you learn from the results of the mining.

So now that you have decided to join the data rush, just how do you avoid stepping in a pile of Big Data? Here are a few common sense suggestions:

1. Know what you are looking for. What question are you trying to answer? Can you convert the effort or the results into something valuable?

2. Have experts who know your minerals and understand how they are used in your industry. Mining is hard and because it is hard it can create value. But would you recognize fool’s gold if you saw it? With Big Data must come a set of key experts who know what real gold looks like, can determine the quality of the gold, and who understand the value of “the find.”

3. Hire a set of good miners and mine operators: people who are experts in gathering the data, storing it, and retaining it in an accessible manner. No one wants a large slagheap of data that obscures the sun.

4. Hire a good set of mineralogists: people who can authenticate the data, who know that it is clean and pure, and who know the value of controlled experiments and methodologies for finding new uses for the minerals mined.

5. Hire the jewelers: people who have the skills to take the raw ingredients of your Big Data minerals and form them into something unique, pretty, and sought after.

The Big Data frontier is getting exponentially bigger every day and we will need to rely on science--data science--to pinpoint the valuable nuggets and avoid the dross so we can find what we are looking for, faster than the competition.

As with any fast-growing field, we will have to learn as we go. We should study the success stories and the failures and adjust our course as needed.

But be assured, “There be gold in them thar piles of BIG DATA!” , so pick up your shovel and get digging.


* DataInformium is a very rare element (I just made up) that can provide useful information directly from raw data without the addition of work.

Please send any comments to