Are you ready for the era of Big Data?

The phrase “Big Data” is now firmly part of the mainstream business lexicon. You hear about it with increasing frequency, but what does it really mean for you and your business? The answer is: a great deal. Big Data not just another lazy buzzword. It has the potential to revolutionize your industry value chain and upend practices that have been in place for years. The revolution has not yet taken root as firmly in many B2B markets as it has among retail consumer-facing enterprises, but that is fast changing. How do you get ahead of the curve and be in the position to profit from the changing role of technology in business decision making? A good place to start is by understanding what Big Data actually is, and how it relates to your company, your customers (and their customers), your suppliers (and their suppliers) and all your other key business partners. Next is to understand what capabilities you currently have and what gaps exist that need to be filled for a full-fledged Big Data capability consistent with the specific characteristics of your demand environment.

How Big is “Big”?

According to a 2011 McKinsey Global Institute research paper, 15 out of the 17 primary industry sectors in the United States store more data per company on average than the entire US Library of Congress (which housed 235 terabytes as of April 2011). For many companies the days of terabytes are long gone and they are dealing in the realm of petabytes (1 petabyte equals 1000 terabytes, and 1 terabyte equals 1000 gigabytes). But the point is not that Big Data kicks in, or starts to matter, at some arbitrary number of terabytes or petabytes. Two things matter more than absolute size when talking about Big Data. The first is that the data come from a great, and growing, number of different sources and need to be managed to be usable. The second is that to actually do anything useful with the data requires powerful cutting-edge scientific methods.

Managing Multiple Data Sources

One of the key distinguishing features of Big Data is that the data come from many different sources. It’s no longer just about the enterprise software database that contains your customer-product transaction history. There are the rapidly proliferating social media sources, real time digital feedback from your sales force, survey-based market research and data feeds providing information on other companies in your industry value chain, among others. All of these data arrive in different formats with varying levels of completeness and integrity. They need to be cleansed, structured and made available for analysis in a way that is intuitive to non-technical business users.

Big Data Needs Big Science

It takes a powerful array of scientific methods to do something useful with all the data – to analyze them, identify patterns, align data-derived recommendations with practical considerations like existing rules, make actionable recommendations, and monitor performance. The vast diversity of insights from all the different sources can enable decision makers to understand their demand environment more clearly, more coherently and at a more penetrating level of detail than ever before. But to do this requires the application of sophisticated quantitative methods like Hierarchical Bayesian models that can overcome problems such as data sparsity, and techniques such as machine learning and neural networks for recognizing and evaluating complex patterns in data. These methods help decision makers fill in the details of the story the data are telling.

Data Transparency Requires Collaboration

Like it or not, the arrival of the Big Data era means that there is far more information out there about everything, including about your company, that is readily and transparently available. This requires a sea change in the way you may have traditionally practiced data management. The value of proprietary data that you keep behind security firewalls and don’t share with your industry partners is diminished in this new environment. This is not to say that data cannot be a competitive advantage for you – it most certainly can. But the key to this success is collaboration, where you work with your upstream and downstream counterparts to fill in the holes and create a more unified picture of your total environment. Retail operators can learn more about the product attributes their customers care about the most, and manufacturers can attain a better sense of the purchasing habits of the people who buy their products. All parties – manufacturers, distributors and retailers – can eliminate waste and improve profitability. Those that intelligently collaborate will benefit at the expense of those who don’t.

How Ready Is Your Company?

There are still many practical obstacles to Big Data – issues related to organizational structures, outdated legacy technology, privacy concerns, shortage of talent with the necessary skill sets – the list could go on. That can lull managers into a false sense of security that change will only happen slowly and thus these are not matters of immediate priority and urgency. A better approach to preparing for Big Data would be to ask the following questions in regard to where your company is situated:

1.    How much value are we generating from the data we take in every day, in terms of deriving enhanced inferences and predictive insights on which we can take action?
2.    Are we using our data in a way that helps us better understand the needs and preferences of our upstream and downstream partners leading to eliminating waste and improving performance?
3.    Do we have the necessary skill sets to make full use of the data, including access to the scientific methods that provide access to patterns and relationships in our demand environment at a very granular, micromarket level?
4.    From #3, what are the gaps we need to fill in order to maximize the potential of Big Data for our business, and where do we go to find solutions that will work for us?

There is no one right solution – every company has its own unique challenges in adapting to a new environment. Asking yourself these questions can help identify the areas where you have potential competitive strengths and the gaps you most urgently need to fill.