#1: Does the solution provide a means for the enterprise to do more with its data than it currently is able to do? How does it deliver this capability, and what makes this approach better than alternatives?

Any business intelligence solution provider is likely to answer the first part of this question in the affirmative: of course we give you the tools to do more with your data. That’s why the second part of the question is such an important follow-up. How does it deliver the capability? What the business user sees when using a BI user interface tool is just the end of a long series of events that starts with the extracting of data from different sources, transforming the data and making it available for the specific discovery, analysis and decision-making activities in which the business users will be engaging. There are a great many decisions and trade-offs that have to be made about the technology that supports these ETL (extract-transform-load) activities. Some choices will make it easier for the vendor to develop and package the solution, but at the expense of the business user’s ease of use. Can your provider show convincing evidence that the technology choices put the interests of the business user ahead of those of the technology developer?

#2: Does the solution provide different stakeholders with a unified picture of demand, at a level of detail to gain a personalized understanding of the preferences, tastes and needs of each customer?

If you’re using business intelligence for better product and customer relationship management then it’s important that they key stakeholders with an interest in the technology will have a single view of the market environment. That’s easier said than done. The problem is not always the massive volume of information, but rather the multiple streams of data that come in from different sources and providers, in different formats and structures. Threading the many strands into a single coherent picture is a significant technology challenge, and you need to be sure that the technology you adopt is up to the challenge. Beyond that, you want to ensure that you will have information at the right level of detail. Beware the “flaw of averages”, where you see information aggregated to a level that loses critical knowledge about the individual customer’s needs and preferences. That loss of knowledge has a measurable cost in money left on the table with each customer-product interaction.

#3: Does it supply the requisite level of domain expertise to provide intuitive business definition addressed to the language, processes and measurement techniques understood by your business users?

Let’s face it: business professionals at any given company, in any given industry sector, speak a particular language. It’s the language of their products, their customers, their processes and their performance measurement techniques. A good business intelligence solution needs to be rooted in a sufficient level of domain expertise to translate the language of the technology into the language of the business. That doesn’t necessarily mean that your BI solution provider has to be a boutique single-industry expert. But it does mean that the core technology capabilities are sufficiently customizable to easily adapt to the functional language of your business environment. Be cautious about off-the-shelf solutions that claim to work as turnkey offerings in any business environment.

#4: Does it allow for rapid time to value without the need for additional investment into technology and resources on your part?

How much pain before the gain? You want to see results early and often, and you don’t want to worry about the need to acquire additional infrastructure or build out an internal IT team to support the technology. In today’s Big Data environment you are probably going to want a hosted solution where the data is warehoused and managed in the cloud. The most important thing to know about a cloud-based solution is that it passes the responsibility of day-to-day management of the data from you to your technology provider, meaning less of an internal administrative/IT burden for you. Then you can focus on the other part of this question: understanding the time frame in which you will be able to see and measure the value delivered by the solution.

#5: Does it adapt and make continuous improvements and facilitate those improvements back into the system for still more informed insights?

The only thing certain about your market environment is that it will change, and often in ways unanticipated when you invest in a new technology. Technology that cannot adapt to new business requirements can go stale very quickly. Does your provider offer a market aware technology? Market awareness is when a technology platform has the capability to learn from new market insights as they evolve, and feed those insights back into the core models and algorithms that power the technology. Market awareness gives you more confidence that your business intelligence solution will be able to rise to the challenges of tomorrow as effectively as it can to the challenges of today.

Knowledge Lost: The Evolution of Mass Marketing

As economic growth gained pace during the Industrial Age this personalized relationship started to change. New modes of production and distribution opened new markets and new customers. Sellers now had to rely on impersonal channels for fulfilling customer demand. In so doing they lost important knowledge about what customers wanted and the value they put on all the different items in their market baskets. They had to rely on second-hand information and gain insights from whatever signals they could detect along the chain of operations from their factories to their sales and distribution agents to their end customers. But these signals and anecdotal bits of  information were often inaccurate, or misleading, or contradictory. Marketing in many ways became a guessing game. Guessing led to bad decisions, unsatisfied customers and money left on the table.

Sales & Marketing in the Early Information Age: The Flaw of Averages

As the Information Age dawned, businesses welcomed the promise of technology in making their operations more effective and productive. But that promise would turn out to be very unevenly distributed across different business activities. Some, like financial reporting or production logistics, readily lent themselves to automating key processes for more operational efficiency. Sales and marketing, though, was a more elusive target whose key performance metrics were harder to quantify.

One of the biggest challenges faced by marketing science during its embryonic years in the 1970s and 1980s was that sellers were now trying to sell the same products to lots of very different types of customers. With the arrival of enterprise technology it was now possible to record those sales electronically and upload them to a data warehouse. Marketing-focused decision support systems were also making their debut at this time. Marketing managers could analyze this intelligence and use it to make decisions around pricing and other marketing levers. But there was a flaw in this approach – call it the “flaw of averages”. The flaw was in aggregating all the data to a top-level average, and substituting that average for the individual tastes and preferences that remained hidden in the details of the data.

Segmentation and the Average Customer

One of the most widely popular responses to these challenges was customer segmentation, an approach that gained popularity over the 1980s and 1990s. Segmentation attempted to group customers into buckets based on a handful of common attributes. But segmentation was not a remedy to the flaw of averages. It still ran on the basis assumption of the “average customer” – and even at the segment level there is no such thing as an average customer. Segmentation did not bring back the knowledge that was lost when marketing ceased to be a highly personalized one-to-one experience.

But not everyone was swimming along with the currents of customer segmentation. In the late 1990s there were small enclaves of engineers and business solutions architects (including some who would go on to be founders of Sentrana) who were exploring a different path: using mathematical approaches like Hierarchical Bayesian modeling to solve problems at the level of the micromarket. “Micromarket” was a concept that had as its basis the town square of old: a venue where buyers and sellers all know each other. The idea was to extract information at this level – for example by the scanner codes employed at individual retail establishments – and apply powerful mathematical modeling to generate insights at that same level of micromarket detail.

Knowledge Regained: Data-driven, Personalized Marketing

This was an approach that could actually solve the flaw of averages. In effect a data-driven micromarket approach can help enterprises regain the knowledge that was lost when marketing left the town square. This is accomplished with information from multiple sources, in multiple formats, at multiple levels of activity to provide enterprises with information about their customers and information about their customers’ customers. That’s information they can not only analyze, but from which they can make decisions, and execute on those decisions, with the ability to make more personalized connections with more customers than ever before.

Sales: A Culture of Personalization

A sales culture has always been built around the core primacy of the personal relationship. Good salespeople are those who know how to manage the personal give-and-take of interactions with their customers in a way that builds loyalty while making that customer’s business profitable for the organization. There are many intangible elements as to what makes this formula work or not work for any given salesperson and that person’s organization. These intangibles make sales one of the most variable activities in the company in terms of performance measurement, with typically a large performance gap between the handful of stars, the large number of average performers and those few at the bottom of the pile. This performance gap is not healthy for the company – it is inefficient and leaves money on the table. A good sales culture is not one where all the value is concentrated in the performance of a few standout salespeople – it is one where the performance of that great bulge in the middle moves closer to that of the sales stars. Here is where data-driven sales & marketing analytics and predictive decision tools can help – not by taking away the personalization of the sales process but by making it stronger.

Personalized Selling Needs Personalized Marketing

While selling has always been about the personal relationship, marketing as it has evolved over the past three or four decades has been anything but. Marketing decisions from pricing to product promotions have moved away from the needs and preferences of individual customers to estimates based on averages for large numbers of customers, typically organized in some kind of segmentation methodology. This means that when salespeople make their calls on individual customers they are not supported by marketing decisions geared to the specific circumstances of each of those clients. That takes away from the personalized value that the salesperson can bring to each meeting.

The arrival of Big Data, rather than being a threat to the personalized sales experience, is a liberator of salespeople from the tyranny of shoehorning customers into predetermined segmentation buckets. Advanced analytics can take the raw data from transaction records and a variety of other sources and transform the data in to insights about individual customer needs and preferences. This can help take a great deal of guesswork out of selling decisions and replace the guesswork with informed guidance that is more likely to resonate with the customer, at attractive terms of offer and at a time when the customer is likely to be receptive to the offer. Salespeople can create impactful customized collateral around these offers that speaks directly to each customer, not to a segment based on some mythical average customer.

Sales and Data: Marrying Two Cultures

Bringing advanced analytics into the organization is not enough – you also have to integrate analytical processes into the selling process. This is a tall order when the organizational culture revolves around traditional notions of sales. For one thing, most salespeople don’t have the time to learn new tools and processes – their hours are more than consumed by the selling and administrative responsibilities they already have. Another obstacle to getting sales buy-in is that people used to viewing success as a result of their unique sales instincts are likely to resist tools that seem to take some of those instincts out of the equation. That reluctance may also be coupled with a concern about what the changes would imply for their compensation expectations, which of course are strongly oriented towards variable performance-based structures.

Quantitative Intuition

In reality it is the human intuition that makes the data useful – without the qualitative insights that the human brain is capable of producing the data tell no story and can serve no role in making better, more profitable decisions. Any reward structure needs to affirm this and make sure the sales representatives understand that compensation is still based on their unique sales abilities. But with quantitative intuition they have the best of both worlds – access to insights they would otherwise not have, the evidence to back those insights up, and knowledge of the odds of those insights leading to new sales. Information and analytics haven’t replaced the sales reps, but rather have made them better.

Are you ready for the era of Big Data?

The phrase “Big Data” is now firmly part of the mainstream business lexicon. You hear about it with increasing frequency, but what does it really mean for you and your business? The answer is: a great deal. Big Data not just another lazy buzzword. It has the potential to revolutionize your industry value chain and upend practices that have been in place for years. The revolution has not yet taken root as firmly in many B2B markets as it has among retail consumer-facing enterprises, but that is fast changing. How do you get ahead of the curve and be in the position to profit from the changing role of technology in business decision making? A good place to start is by understanding what Big Data actually is, and how it relates to your company, your customers (and their customers), your suppliers (and their suppliers) and all your other key business partners. Next is to understand what capabilities you currently have and what gaps exist that need to be filled for a full-fledged Big Data capability consistent with the specific characteristics of your demand environment.

How Big is “Big”?

According to a 2011 McKinsey Global Institute research paper, 15 out of the 17 primary industry sectors in the United States store more data per company on average than the entire US Library of Congress (which housed 235 terabytes as of April 2011). For many companies the days of terabytes are long gone and they are dealing in the realm of petabytes (1 petabyte equals 1000 terabytes, and 1 terabyte equals 1000 gigabytes). But the point is not that Big Data kicks in, or starts to matter, at some arbitrary number of terabytes or petabytes. Two things matter more than absolute size when talking about Big Data. The first is that the data come from a great, and growing, number of different sources and need to be managed to be usable. The second is that to actually do anything useful with the data requires powerful cutting-edge scientific methods.

Managing Multiple Data Sources

One of the key distinguishing features of Big Data is that the data come from many different sources. It’s no longer just about the enterprise software database that contains your customer-product transaction history. There are the rapidly proliferating social media sources, real time digital feedback from your sales force, survey-based market research and data feeds providing information on other companies in your industry value chain, among others. All of these data arrive in different formats with varying levels of completeness and integrity. They need to be cleansed, structured and made available for analysis in a way that is intuitive to non-technical business users.

Big Data Needs Big Science

It takes a powerful array of scientific methods to do something useful with all the data – to analyze them, identify patterns, align data-derived recommendations with practical considerations like existing rules, make actionable recommendations, and monitor performance. The vast diversity of insights from all the different sources can enable decision makers to understand their demand environment more clearly, more coherently and at a more penetrating level of detail than ever before. But to do this requires the application of sophisticated quantitative methods like Hierarchical Bayesian models that can overcome problems such as data sparsity, and techniques such as machine learning and neural networks for recognizing and evaluating complex patterns in data. These methods help decision makers fill in the details of the story the data are telling.

Data Transparency Requires Collaboration

Like it or not, the arrival of the Big Data era means that there is far more information out there about everything, including about your company, that is readily and transparently available. This requires a sea change in the way you may have traditionally practiced data management. The value of proprietary data that you keep behind security firewalls and don’t share with your industry partners is diminished in this new environment. This is not to say that data cannot be a competitive advantage for you – it most certainly can. But the key to this success is collaboration, where you work with your upstream and downstream counterparts to fill in the holes and create a more unified picture of your total environment. Retail operators can learn more about the product attributes their customers care about the most, and manufacturers can attain a better sense of the purchasing habits of the people who buy their products. All parties – manufacturers, distributors and retailers – can eliminate waste and improve profitability. Those that intelligently collaborate will benefit at the expense of those who don’t.

How Ready Is Your Company?

There are still many practical obstacles to Big Data – issues related to organizational structures, outdated legacy technology, privacy concerns, shortage of talent with the necessary skill sets – the list could go on. That can lull managers into a false sense of security that change will only happen slowly and thus these are not matters of immediate priority and urgency. A better approach to preparing for Big Data would be to ask the following questions in regard to where your company is situated:

1.    How much value are we generating from the data we take in every day, in terms of deriving enhanced inferences and predictive insights on which we can take action?
2.    Are we using our data in a way that helps us better understand the needs and preferences of our upstream and downstream partners leading to eliminating waste and improving performance?
3.    Do we have the necessary skill sets to make full use of the data, including access to the scientific methods that provide access to patterns and relationships in our demand environment at a very granular, micromarket level?
4.    From #3, what are the gaps we need to fill in order to maximize the potential of Big Data for our business, and where do we go to find solutions that will work for us?

There is no one right solution – every company has its own unique challenges in adapting to a new environment. Asking yourself these questions can help identify the areas where you have potential competitive strengths and the gaps you most urgently need to fill.

your demand environment doesn't stop with your immediate customers

How would you describe your demand environment? How would you define its boundaries?

Managers in business-to-business (B2B) enterprises typically respond to these kinds of questions by referencing a host of familiar business practice models articulated and implemented over the last 20-30 years. According to these practices “demand” involves everything that takes place between you and the customers who buy things from you, while “supply” refers in turn to the mechanics of everything you procure from upstream vendors. Demand is the world of sales & marketing, of pricing and running promotional campaigns and figuring out what products to bundle with others to win a bigger share of the customer’s total market basket. Supply is operations and logistics, inventory cost management and procurement processes. The concept of a “supply chain” has taken firm root over the last decade or so, while demand is seen less as a sequential chain than as a loose collection of activities organized around a point of contact between the enterprise and the collective needs and preferences of the customers who directly buy its products and services. The “art of the sale” is thought to be a less quantifiable notion than the “science of logistics”, with a greater X-factor that does not easily lend itself to data-driven analysis. This is the traditional view – but managers today are faced with a new set of realities that require a different way of looking at the enterprise’s demand environment.

One Person’s Supply is Another’s Demand

Increasingly a B2B company’s demand environment is impacted by, not only what its own customers are demanding, but what their customers are demanding, all the way down to the patron who visits a retail outlet or sits at a restaurant table. On the other side, one business’s supplier is another business’s customer, and so on all the way up to the cornfields and copper mines that supply basic ingredients and raw materials to the production and distribution process. Information flows more freely throughout this ecosystem, and is more transparent and widely available from multiple sources, including some that didn’t even exist a decade ago. At the same time, businesses are under constant pressure to deliver profit and margin performance in an era of modest prospects for top-line growth. This requires managers to find new ways to make their operational processes just a bit better – to eliminate waste wherever possible.  This in turn requires decision makers to extend their view of demand the beyond the traditional boundaries of their immediate customers to incorporate an end to end chain.

Here are three examples of waste that occur at different points along the demand chain. By adopting the end-to-end demand chain view it is possible to see these as three variations of a common problem rather than three separate problems, and to solve them via common, collaborative, data-driven solutions.

#1: Manufacturers Waste Millions in Trade Spend

For manufacturers in many industries two of the most profit-draining words in the English language are “trade spend”. Increasingly trade spend has become a pay-to-play proposition, and involves cutting a check to one’s distribution partners with little if any knowledge of how effectively (if at all) that money is being applied to increase sales of the manufacturer’s products. More importantly, these trade spend dollars provide no useful insights into customer purchasing habits – who is buying which of your products, where, when and in what quantities. Practices are notoriously non-automated and vulnerable to human error.  Manufacturers experience waste because they do not have adequate visibility into what is happening downstream, and lack the tools to turn trade spend into trade investment.

#2: Distributors Waste Millions in Sales & Marketing Guesswork

For distributors, whose core business proposition is selling, the oft-repeated mantra sounds simple enough: the right product (or assortment of products) to the right customer at the right time for the right price. That “right” price is the highest price obtainable by the seller that will at the same time be seen as fair and retain the customer’s loyalty. More often than not, though, this decision is made on the basis of guesswork more than on the basis of empirical evidence. Large distributors have hundreds of thousands of customers and products, which implies tens of billions of customer-product combinations. Each of these combinations potentially is driven by unique factors that influence what the optimal terms of offer should be. Distributors experience waste when they lack the ability to put these factors into perspective that lead to the right price, promotion, assortment and timing decisions.

#3: Retailers Waste Millions in Suboptimal Ordering Mechanisms

The “bullwhip effect”, also known as demand whiplash, is well known in industry ecosystems. As retailers incorporate changing demand patterns into their forecasts and transmit these new data signals upstream, their upstream partners successively adjust their own forecasts. The result, like a game of telephone tag, is that information at the end of the chain has deviated wildly from what it was at the beginning. By the time orders make their way back downstream the retailer operator’s environment will have changed again, resulting in either costly inventory build-up or even more costly out-of-stock situations. Retailers experience waste when they lack effective and responsive ordering mechanisms with their distribution and manufacturing partners.

Collaboration: Solving Different Sides of the Same Problem

The problems manufacturers, distributors and retail operators experience in demand chain waste lend themselves to finding common solutions through collaboration. By understanding how problems at different flashpoints upstream and downstream impact each business regardless of their location, demand chain partners can contribute their respective areas of insight and expertise to reduce waste across the chain to benefit all parties. For example, much of the guesswork distributors deal with comes from a lack of detailed knowledge about the product attributes that resonate with specific target customers. Manufacturers can contribute their deep product knowledge to overlay and enhance what the distributors already know about their customers’ buying habits, giving them an additional edge in knowing what categories and products to target to what audiences for optimal effect. In turn distributors can pass these insights directly downstream, targeting customers with attractive offers and supporting collateral for the products they are most likely to be in need of at specific times in the calendar year and making them available via efficient ordering systems that minimize sales channel waste.

In a demand chain world the fortunes of upstream, midstream and downstream players are much more interdependent than they used to be. In this environment demand does not stop at the point of contact between a business and its immediate customer. A collaborative approach to improving efficiency and reducing waste can help improve the financial performance of all parties.

Big Data presents challenges for business managers and IT pros alike

The Big Data revolution taking place across many industry sectors is presenting challenges to business managers in traditional line functions like production and marketing, and to corporate IT professionals alike. Big Data goes to the heart of the IT professional’s domain – the way that the enterprise warehouses and manages information – and potentially upends long-established systems and practices. At the same time it presents business manages with a new level of urgency to incorporating deep data analytics into their daily array of tactical business decisions. As important as it has always been for business and technology professionals to be on the same page, it is now more important than ever. Both sides also need to understand the specific concerns and challenges that are keeping the other up at night, and work together to solve them and identify critical gaps.

Not Just Rows and Columns Anymore

The world of the corporate IT professional has long been an orderly environment of rows and columns – the architectural foundation of relational databases containing daily transaction activity. This is not to say that the job was ever easy – there are many things that can compromise the integrity and accuracy of the information residing in those rows and columns. But at least the mission was clear – figure out the best way to warehouse the data, scale the size of the warehouse to accommodate the growth in data volume, and make clean, usable data available to the business user community every day. The key to this was structure – the data in this closed, managed environment were structured and supported by the technology systems in place. That structure is no longer applicable to all of the data coming into the company’s technology environment every day – in fact, a large part of the growth in data volume is coming from other sources that are decidedly unstructured.

Rise of the Machines

What is coming into corporate environments with a higher frequency now are open source data in many new formats that evolve rapidly and scale even more rapidly. In 2011 the amount of data created and replicated around the world exceeded 1.8 zettabytes – that’s 1.8 trillion gigabytes, representing a growth factor of nine times over the previous five years. The really daunting challenge facing IT professionals as they contemplate this staggering growth rate of information is that most of the growth is coming from various types of machine generated data (MGD), i.e. data generated automatically by an independent computational agent that is not caused by human action. Clickstream logs, social media, stock exchange trading data are all examples of MGD.

Chances are that sources of MGD data exist in just about every corner of major US industry sector. Where do all these types of data come from? According to a 2011 white paper by telecommunications company Ericsson, there are expected to be over 50 billion connected devices in the world by 2021, each of which will be continually emitting data-rich signals. Buried within these seemingly unintelligible signals are data about consumer behavior, location, transportation methods and other insights that can provide business managers with a fuller, deeper picture of their demand environments. The challenge of managing this information goes well beyond traditional challenge of uploading and warehousing the data in the most optimal way for the business users to extract and analyze them.

It’s Not the Data, It’s What You Do With the Data

Making sense of this tidal wave of multi-source and multi-format data will be challenging; yet there are some basic rules business managers can follow to keep themselves from becoming overwhelmed. Any data – whether from a traditional proprietary transaction database or from an RFID scanner or from social network weblog files – are useful only to the extent that human intelligence enters the picture and asks the right questions from which to derive actionable insights from them. For example, your raw data may tell you that one of your customers – a restaurant that buys food products from you – tends to buy a preponderance of raw ingredients while another customer – a restaurant of a similar type – buys mostly prepared foods. That basic information leads to a derived data point that you can call something like “chef skill level”, the idea being that the restaurant offering menu items prepared from scratch places a comparatively high premium on the quality of their dishes and invests in chefs of commensurate experience. That data point in turn can lead to other predictive insights, such as which of these two customers would be more receptive to a particular product promotion or a loyalty incentive discount.

Source of Competitive Advantage?

Can information management practices in the age of Big Data offer a path to competitive advantage? There’s plenty of reason to be skeptical – there is no shortage of historical cases where emerging technologies have been trumpeted as the Holy Grail, and the vast majority of such claims have failed to ring true. Big Data can sound like little more than hype – but there are stronger reasons to argue that it is real, and here to stay. Achieving sustainable advantage for an enterprise requires an intelligent approach to managing and using information based on the following:

•    Coordination between suppliers, distributors and retail operators to reduce waste along their demand chain and realize opportunities for mutual profit;
•    Connection to all the new sources of data emerging into the ecosystem, and the ability to take action on insights from the data in real time;
•   Customization of offers that provide the right product (or bundle of products) to the right customer at the right time, at a price that maximizes the seller’s profit while retaining the buyer’s loyalty.

Whether one believes that Big Data represents an evolution or a revolution, it is still new enough to be a large question mark in the minds of many business decision makers. The opportunity is present for visionary enterprises to establish a market driving position; but that opportunity won’t last forever.

what matters is understanding the concept, not the formula

Causation Fables

If you have ever tuned into the financial markets report on the evening news (or switched on CNBC just about any time on any given day) you are no doubt familiar with the formula by which the day’s action is delivered, with the anchor saying something along the lines of “stocks fell today on news of higher home foreclosures”. To drive the point home the broadcast will trot out the stock footage of suburban homes with foreclosure signs in the front yard. These kinds of reports are snappy and speak to our very human need to supply a causation narrative to the events we encounter in our lives. Small wonder that the question of causation has beguiled and tormented philosophers since the beginning of human civilization.  

These kinds of reports are also, however, misguided. Economic outcomes like stock prices or consumer choices are not “caused” by any one single thing; rather, they are influenced by a variety of different things on any particular day. The relationship between an outcome and anything that exerts an influence on it is expressed by the statistical concept of correlation. Now, it is probably understandable why the news anchor with 20 seconds to convey what happened on Wall Street today uses the convenient language of causation. “Increased housing foreclosures caused today’s fall” is simply more digestible than “stock prices tend to exhibit negative correlation with changes in home foreclosure levels, and that tendency was on display again today”. But business managers in sales & marketing do not have the same luxury as news anchors. They need to grasp the difference between causation and correlation, and understand the dynamics at play with all the things that influence behavior in their demand environments.

Marketing Needs Analytics

Historically, sales & marketing decisions have been made largely by means other than the close analysis of massive amounts of data, and managers making those decisions have relied more on guesswork than on empirical evidence. In other words simplistic causation narratives have trumped more granular correlation-based analysis. “Lowering prices will drive up sales” and “people will buy more of our products during the busy season” are two particularly well-loved narratives. On the face of it these causation stories appear reasonable enough. If you observe often enough that sales go up when prices go down, or that people buy more eggnog mix in the month of December than any other time of the year, then it seems logical to make the cause-effect link.

But the reality is far more complicated. How many more cases of a given product do people want to buy for each unit of price reduction? Which people, specifically, are more responsive to the price move? How does the relationship between price and quantity vary among products in a given category, and among the customer types who buy those products? What influence on the relationship between price and quantity can be attributed to other marketing decisions – like impactful promotional campaigns, cross-selling and calendar planning – as well as to things that are beyond the control of the seller, like competitive actions or acts of nature? What is the highest price the seller can charge without losing the customer’s loyalty?

Enter Correlation Analysis

These are not questions that lend themselves to easy answers, or to simple guesswork based on observed cause-effect relationships. This is where correlation analysis steps in. Correlation measures the strength and direction (i.e. positive or inverse) of the relationship between variables, and the extent to which one variable can serve as a prediction of the outcome of another. For the types of questions posed in the previous paragraph, correlation analysis is supplying quantitatively supported answers in the form of explaining how much influence any one thing has – unit price reductions, promotional campaigns, expansion of a competitor’s sales territory – on the quantity of specific products that specific customers elect to purchase.

To provide meaningful correlation-based insights in the highly complex demand environments of today’s economy requires sophisticated mathematical modeling techniques capable of measuring, not just the relationship between two isolated variables but of multiple variables together. In its simplest form correlation is a linear relationship between two things – say, a time series showing the price charged for a certain product and the quantity demanded at that price. The closer the data points fall along a line of best fit drawn through them, the more explanatory power (i.e. the higher correlation) the relationship is said to have. But to provide a comprehensive picture of all the principal factors influencing demand, the models have to be able to integrate all those pairwise relationships into a coherent, unified view accessible to all decision makers involved with the execution of sales & marketing tactics.

Business enterprises today are faced with the fundamental challenges of slow growth and constant pressure on profit margins. In this environment every insight that can lead to better marketing decisions – from knowing the optimal price for a given product-customer opportunity to knowing when to time a promotion for the best return on investment – is an insight that managers cannot afford to pass up. Teasing out these insights requires data-driven techniques for understanding how the different factors at play in your demand environment are correlated, and what you can do to improve the likelihood of achieving the best terms for each potential sale.

there are no average customers

Imagine that you have two customers to whom you are selling beef. One is a greasy-spoon hamburger joint and the other is an upscale French bistro that offers a pricey version of steak tartare as one of its gourmet entrees. In figuring out the terms of offer for each restaurant, would you create a profile of an “average customer” based on these two data points and sell a single variation of medium-grade beef at a mid-range price to both the burger joint and the classy bistro? Of course not – you know perfectly well that approach would please nobody and result in two lost sales. Yet that is exactly what sellers in many industries are doing every day – marketing to some notional “average” customer that in reality does not exist. It may seem less absurd when the average is based on thousands or tens of thousands of customers as opposed to two – but it is no less flawed as a marketing approach. Every single customer has a unique set of needs, preferences and priorities. It is your job as a seller to tailor your offers as close as possible to each unique demand curve. Fortunately, technology and science make that daunting challenge possible.  

Segmentation – Top-down Guesses

Of course, marketers realized years ago that customers are not all the same. They created a way to incorporate this realization into their decision making, which we all know today as customer segmentation. Segmentation can make average customer profiles seem somewhat more reasonable than lumping all customers into one bucket – for example, one could reasonably assume that the hamburger joint and the French bistro would not wind up in the same segment. But ultimately, segmentation is about foisting top-down guesses about what are meaningful differentiators onto a universe of customers. In the early days of segmentation the idea was to delineate segments by zip code, assuming that the economic characteristics of a customer’s place of residence were a good indicator of his or her likely spending habits. This became popularly known as demographics. In the 1980s “psychographics” became the in-vogue approach to segmentation, for better or worse bequeathing to us “chardonnay-drinking Volvo-driving NPR-listening urban dwellers” and the like. Again, psychographic categories represent guesses about the attributes likely to correlate to specific consumer tastes and preferences.

Flaw of Averages

What demographics, psychographics and any other kind of -graphics have in common is that someone dreamed them up as a theory about what drives customer behavior and then used the theory to make determinations about how to market more effectively to their customers. But ultimately with this approach you are still dealing with flawed assumptions about the needs and preferences of your actual customers. Rather than denoting your entire customer base with one Average Customer you now have one Average Customer for each segment. Put another way, not every “Chardonnay-sipping Volvo driver who lives in zip code 06830 (Greenwich, CT)” is the same person with the same needs and preferences.   These top-down assumptions cannot begin to untangle the specific multitude of factors that drive individual customers to particular categories and specific product SKUs within those categories, let alone the more nuanced considerations of what messages they are likely to respond to, when they are likely to be responsive and what other products they might be persuaded to buy in the same time frame.

Let the Data Do the Talking

A better approach than top-down guesswork is a bottom-up approach that lets the data speak for themselves. The insights are there in the transaction records that come into your data environment every evening, and increasingly from other sources as well such as social media networks, surveys and third party data providers. You do not need to embellish these with guesses about the best way to segment your customers. What you need is a rigorous methodology for formatting, analyzing and acting on the insights the data provide. What are the factors to which certain customers respond? What like tendencies may be seen between different types of customers, and why? What products are your customers buying from other suppliers that they could be buying from you, if suitably induced? Let the data reveal these patterns through applied scientific methods, rather than making top-down assumptions about which attributes go with which customer segments. You may wind up knowing your customers’ buying habits better than they themselves do.

When are your customers most likely to change their buying habits?

Habits and Decisions

Habits are much talked about in sales and marketing circles these days. In a recently released book “The Power of Habit: Why We Do What We Do in Life and Business” New York Times reporter Charles Duhigg approaches this subject from the interaction between habits, decision making and the human brain. Our brains are constantly working to convert active decision making exercises into habits, writes Duhigg, because it conserves energy to do so. Decision making requires a great expenditure of time and mental effort. Even what we think of as relatively simple activities, such as backing the car down the driveway, require the fine-tuned calibration of many complex parallel processes running in the brain (think about the first time you actually backed a car down your driveway, before it became an ingrained habit). In the world of business this argument extends to the way people make purchasing decisions – in short, wherever possible we convert the energy of active decision making into passive habit. Once comfortably set in these habits we are reluctant to change them.

Breaking the Habit

With this in mind we come back to the problem of timing when we go out to customers with enticements to get them to buy products from us that they are currently buying elsewhere. We have to assume that these customers are comfortably set in their current purchasing habits and therefore that it will be hard to dislodge them under most circumstances, even with incentives like price discounts. We look to the data to see if there are clues as to when the optimal time might be. In other words, for any targeted product promotion to a particular customer we try to identify a set of factors than can best predict when the customer is likely to be most open to changing his or her current purchasing habits.

Seasonality – It’s Not that Simple

Seasonality may play a role – many products are seasonal by nature and purchasing incidence will logically be brisker during the peak season. Does that mean that the height of the busy season is the right time to launch the promotion? Perhaps it does not mean that at all. Think again about habits and decisions. Better yet, put yourself in the shoes of a purchasing manager for a catering business that specializes in holiday events. You can imagine that in the week before Thanksgiving this purchasing manager is buying lots of cranberry sauce. How receptive is she going to be to a promotion you launch at this time, with price discounts and other incentives to purchase cranberry sauce from you?

Before you say “very receptive” and rush to launch a cranberry sauce promotion, think about what’s going on in her life. What this purchasing manager probably cares about more than anything else right now is having lots of cranberry sauce and other seasonal items stocked on the shelves so that there will not be a shortage on Thanksgiving Day when they will be rushing around from one catering venue to the next. Put another way, she does not want to have to think about whether she is getting the best deal from her current supplier – she knows that in the past the supplier has met the organization’s needs to have plenty of cranberry sauce on hand. Using the same supplier this year is just one less thing to worry about during a time when plenty of other stressful decisions will be placing demands on her brain cells. So in fact it may be a very bad time for this promotion.

Let the Data do the Talking

If this is true, then how can you make smart decisions about promotional timing other than relying on lucky guesswork? At Sentrana we believe that the best way to approach questions of timing is from the ground up – let the data point the way rather than imposing a preconceived intuition into the model. In situations similar to that of the hypothetical purchasing manager in the previous paragraph, we have discovered a tendency for first-time purchases (i.e. when a customer buys a product from a supplier that he previously obtained elsewhere) to occur around two months before peak seasonality. So going back to the previous example, if you were to approach the catering purchasing manager in September with a promotional campaign for cranberry sauce, including an introductory price discount and some well-crafted supporting collateral (such as recipes or suggested uses) then you may well get her attention and be better positioned for a successful outcome.

The data are out there – you have extensive historical sales records from which to obtain useful insights about things like the relationship between first-time purchases and seasonality peaks. Let the data do the talking, and focus your own efforts on connecting the dots that will help you build a promotional calendar with a higher likelihood of aligning the right products to the right customers at the right time.

certain products go together

At Sentrana we believe that companies can increase sales, even in tough economies, by understanding their own demand environments at the most detailed level possible – in other words, to be able to predict what products to offer to what customers, and to use insights from available sales data to make targeted recommendations around pricing, promotional activities and timing. In foodservice hundreds of thousands of products pass through any given distribution channel every day to hundreds of thousands of restaurants and other operators. To meet this challenge effectively manufacturers and distributors need to contribute their respective insights about products and customers onto a common platform from which to obtain a full picture of demand. Recently this has motivated prominent industry players to collaborate in managing performance across key product categories.

Despite these different goals there is a way for category management to lead both manufacturers and distributors to direct financial benefits, not merely demand shift. Consider the case of tomato sauce we used as an example above. Now, at any point in time a single manufacturer – call it Manufacturer A – has a certain market share for each product it sells. The end customer – the foodservice operator – may be buying Manufacturer A’s brand or it may be buying a competing brand. Over any defined market (e.g. regional sales territory) the incidence of purchase of Manufacturer A’s brand should be equal to this manufacturer’s share of the market.

Let’s focus first on what is happening at the distributor level. The distributor’s goal – call it Distributor A – in this scenario is to create conditions by which an end customer will want to buy a certain product from Distributor A that the customer now buys from somewhere else. That is understandable in the abstract, but in the real world how is Distributor A supposed to know which customer to approach, which product to offer, and the terms at which to make the offer such that it will be attractive to the customer to shift purchase?

The answer to this involves a technical term – association and classification modeling – and a more reader-friendly explanation: certain products go together. The distributor’s sales data may identify 100 customers who have recently purchased prepared pizza crusts, tomato sauce and mozzarella. If the 101st customer recently purchased pizza crusts and mozzarella, it is a reasonable prediction that the customer is purchasing tomato sauce from somewhere else. The models we referred to above spot this opportunity and alert the relevant decision makers. We have homed in on which product to offer to which customer.

We still have a problem, though. We have identified the opportunity at the product level – tomato sauce – but do we know enough about the customer to understand his or her preferences within that product area? From the distributor’s perspective the answer is probably: no. The distributor’s job is to move product, not to be deeply familiar with the qualities and attributes of individual brands and SKUs. So now we must move the focus upstream to the manufacturer, who does possess that deep brand knowledge. Manufacturer A can tell us what product attributes may be most attractive to the customer to whom we are trying to sell the tomato sauce. This helps Distributor A move to a further level of granularity and identify which SKU/s, out of all the possibly hundreds that exist in the tomato sauce classification, may be the most likely to induce the customer to switch from their present distributor. Manufacturer A can even provide supporting sales collateral like recipes and usage suggestions to help Distributor A’s sales representatives close the deal.

Now we come to the real value proposition for the manufacturer. What has transpired in the scenario we described above is that a sale of any tomato sauce by any distributor has become a sale of a specific tomato sauce SKU to a deliberately targeted customer. The sale of “any tomato sauce” may have involved one of Manufacturer A’s brands or it may have involved a competitor’s brand – in aggregate, as noted above, this would be in proportion to Manufacturer A’s market share. For every instance where the customer would otherwise have purchased a competing brand, the sale of a targeted SKU through Distributor A results in incremental sales growth for Manufacturer A. Not demand shift, but real incremental growth.