The Importance of Collaboration

In foodservice, manufacturers and distributors are the logical partners for a collaborative category management venture. Manufacturers possess deep insights into the product attributes that drive demand for specific customer types, and have a strong understanding of how to manage assortment. On the other side, distributors have the benefit of daily transaction data at a very granular level – what quantities of products in the category are being sold to what locations with what frequency. Combining these insights – ideally through a single integrated data management system able to process inputs from multiple sources and generate insights and actionable recommendations to the relevant decision makers – can create a coherent, unified picture of demand that provides a basis for specific assortment, pricing and promotional activities to grow sales.

Reducing the Guess Factor

A traditional SKU rationalization program may analyze aggregate transaction histories for all the SKUs in a category and mark for elimination some subset of those that occupy the so-called “long tail” – products with sparse data records due to infrequent activity. A typical goal in this regard may be to eliminate 20-25% of all SKUs in the category. The problem with this approach is that without an appropriately detailed level of analytical insight, managers are left to guessing what the resulting effects will be on sales. Transaction frequency is only one variable in presenting a composite picture of demand. For example a certain product may transact on an infrequent basis only, but it may also be a popular niche product with attributes highly valued by major customers. What will the sales impact be of not having this niche product available when a major customer wants to add it to his or her market basket? How can decision makers recognize and differentiate between niche products and other long tail denizens that really deserve to be eliminated from the active product line?

A holistic category management solution, driven by advanced predictive science, can supply answers to these questions. By integrating product attribute knowledge possessed by the manufacturer with quantity and purchase timing data known by the distributor, the system can make recommendations about when to stock the low-frequency but desirable niche items with a higher likelihood of coincidence with the customer’s purchase decision. Techniques such as Hierarchical Bayesian modeling help overcome the analytical challenges typically presented by sparse data. Rather than losing all or part of the customer’s market basket for the sake of an incremental SKU reduction – in most likelihood a losing proposition – the result is retaining a satisfied customer.

Focus on Growing Demand

This approach to category management program shifts attention away from simple cost reduction through inventory rationalization and focuses instead on the revenue side of the equation – growing demand in the category. There are two critical requirements for this to be successful. First, the data management platform must be sufficiently granular to provide meaningful insights at the level of every customer and every product (for example as in the long tail analysis described above). Second, the platform must seamlessly transform into a practical tool which sales representatives can use in the field. This is a particularly important requirement. Foodservice sales & marketing representatives as a rule have very little time for incremental effort above and beyond their existing selling and administrative responsibilities. They certainly do not have sufficient time to juggle multiple sales tools offered by multiple manufacturers acting in the role of category manager. The ideal tool is one with which the representatives have existing familiarity (to avoid time-consuming learning curves for new processes) and which can seamlessly integrate data from multiple input sources.

Manufacturers and distributors need more than just a rationalization program to optimize performance at the category level. A holistic approach, supported by robust analytics delivering actionable real-time guidance to sales professionals in the field, can improve category performance all along the foodservice value chain.

So you have been asked by your most important foodservice distribution partner to be a category captain. What happens next? As captain you are tasked with managing the assigned category for optimal performance. That entails the following:

•    Analyze all products across the category (not just your own brands)
•    Augment the data provided by the distribution partner with your own internally generated insights
•    Provide structured, actionable recommendations based on intelligence obtained from the data

These recommendations relate to product assortment, pricing policies, promotional activities and other important demand levers for driving profitability. At the same time you need to educate your distribution partners, both at corporate headquarters and in the field, about the product characteristics that can help increase demand. This requires an intelligent approach to data analytics.

What insights about products can help drive category sales?

What might a good analytics model for category management look like? Let’s consider the key tasks we identified in the previous paragraph.

Analyze All Products Across Category

Category management is driven by analytics. As category captain you will receive transaction data from your distribution partner to form the basis of your insights and recommendations. The first issue with which you will likely have to deal is the quality and completeness of this information. Bear in mind that foodservice distributors are typically not used to sharing sensitive sales data with their suppliers, and may lack effective internal processes for making it available. Robust data management solutions like Sentrana’s MarketMover™ help collaborative category partners overcome this challenge by providing timely access to clean, customer-level data.

The next order of business is to map out the analytical processes that can best support your distribution partner’s objective to improve category demand. This may be best approached through posing a series of questions. For example:

•    What intelligence can we derive from the data to help identify ways to improve product demand among existing customers?
•    What patterns and associations will provide us insights about products that current customers are not buying from our distribution partner but could be enticed to buy?
•    How can we improve sales turnover by encouraging customers to switch from lower-to higher-velocity SKUs?

Augmenting Data with Internally Generated Insights

In answering those and similar questions one of your most important activities is to augment the data your distribution partner provides with your own unique insights about the products in the category. An important example of this are the product attributes that drive demand among certain customer types. Perhaps you are charged with managing baked goods and you need to figure out what the right use of shelf space will be for muffin products. Your distribution partner’s objective is to increase total muffin sales – for example along the lines of one of those three questions posed in the previous paragraph. As a manufacturer you can provide your own deep knowledge about what features and attributes drive sales among certain customers.

A critical data challenge, therefore, is to have the ability to map specific attributes to specific products. Category managers should be able to access the product database and establish product groupings and categories based on like attributes. Using the above example, for every product you can assign a  quantitative attribute metric. “Butteriness” may be an appropriate attribute for muffins, and you can rank all applicable products along the lines of “very / moderately / not very buttery”. This can facilitate more rational product groupings within the category that better enables you to analyze and evaluate assortment trade-offs, pricing strategies and promotional approaches.

This kind of product administration capability brings up in turn a whole series of issues around how to create standardized attribute definitions for each relevant subcategory and product set. By allowing category mangers to create new product and subcategory groupings, it becomes likely that these categories will not map directly to those of the distributor. A category administration functionality is required that will manage the interface between the distribution taxonomy and the specific product and attribute groupings mapped by category managers at the manufacturer.

Supporting Analytics

As you map out these processes you can get a better sense of what the analytical capabilities may look like. For example, what data exploration functionalities can help you analyze effectively? To orient your own understanding of the structure of subcategories and products to its organization in your distribution partner’s data records you need a mechanism for guided drill-down and drill-up within products, as well as in regard to customers, sales territories and other key information. There should be a filtering mechanism that allows you to view products along a number of descriptors: for example, all the SKUs currently stocked at a particular operating company of your distribution partner, or all products with the item description “frozen”.

Visualization and editing capabilities are also important components of analysis. How do you want to see the information and organize it into compelling formats for your targeted readers? You will probably want to have a variety of formats and the tools to manipulate data into different visual representations to underscore the insights about customers and products you wish to communicate. You will also want to be able to easily access and modify the reports and formats you use most frequently, and to share reporting and editing capabilities with others working on the same projects.

Providing Guidance and Recommendations

Category managers need to consider how best to translate analytical insights into actionable recommendations for their partners. For example, developing strong promotional content around products for the distributor’s sales force can be an important way to execute against category performance targets. A system for uploading, managing and exporting product-related content is thus an important functionality to consider. Another valuable feature could be scenario analysis capabilities to map out alternative approaches to pricing decisions, promotional opportunities and assortment trade-offs. Finally, manufacturers need to consider how to incorporate data from their own market sources: for example sales information at the total market level rather than just the share occupied by their distribution partner.

Collaborative category management can evolve into a long-term relationship that will improve category performance for distributors and improve overall product sales for manufacturers. Over time the scope of a category management program may expand to include enhanced predictive initiatives and a fuller set of demand levers. Building a good foundation with the right data analytics is a good place to start.

intelligence on competitors' prices may be close at hand

If only you knew what your competitors are charging. How many times on any given day does that phrase get uttered in a corporate boardroom, on a sales call or in a marketing strategy meeting? Knowing what your competition is charging would take so much of the guesswork out of your daily pricing and marketing decisions. It may come as a surprise, then, that critical information capable of revealing competitors’ prices may be very close at hand – in your own purchase history.

Is there a “Market Price” for COGS?

Since you do not have direct access to your customers’ prices, the challenge is to model likely competitor activity based on incomplete information. A good place to start is with costs – specifically cost of goods sold (COGS). This is typically a key input in pricing models. Knowing a competitor’s COGS would provide critical intelligence in determining what prices they are offering in the market.  The trick is to accurately infer the “market” price a competitor pays for their inputs (i.e. their COGS) from the information contained in your own transaction data.

Birds of a Feather…

Advanced analytics can help in this regard by identifying patterns, or correlations, between similar products within a category. Individually, of course, the price of a commodity like milk or sugar will fluctuate over time. However, like products tend to move together. For example, you may find that within the  dairy category products like milk, butter, cream and cheese all tend to follow the same pattern in price movement.  Whether prices go up or down, they tend to do so with the same magnitude and frequency. Therefore, it is possible to infer an expected market cost for each of these products by observing their price correlations.

Spot the Outlier

If we are able to form confident expectations about the price movements of key COGS inputs from correlation patterns, then we should be able to flag anomalies – deviations from our expectations that may tell us whether our cost assumptions about a certain product are out of line with the “market”.  For example, we may notice that the price of milk has increased by 1% whereas the prices of butter, cream and cheese have all gone up by 2%.  We may further infer that our competitors are paying this “market” price for milk and adjusting prices to their own customers accordingly. This in turn, allows us to compare our own COGS to the expected market price and adjust our customer pricing to win additional business or capture greater value without incurring additional risk.

The margin for error in foodservice is narrow. Price too low and you risk sustaining margin erosion. Price too high and you increase the chance of losing a sale to a competitor. Having the information at hand to infer what your competitors are factoring into their pricing models will put you in a stronger position to set your own offers at the optimal level.

Elegance does not always lead to the best outcomes

Elegance in modeling is expressed in the appearance of simplicity – rendering sprawlingly complex interrelationships in the real word into the clarity of precise mathematical formulae. Simplicity and elegance are all well and good, unless in the quest for this holy grail you wind up dramatically misrepresenting how things actually work in the environment you are trying to model. This can result in not only failing to solve the business problem at hand, but actually making matters worse than status quo ante by facilitating decisions based on incorrect assumptions. We have a real world example of just how much worse this can be in the financial markets debacle of 2008, when the elegant models crafted by the best and brightest quantitative experts Wall Street had to offer proved to be fatally flawed in the assumptions and heuristics they used to express the variables affecting housing prices, interest rates and mortgage payment trends. Perhaps modelers need to live by something like the Hippocratic oath taken by medical doctors: first of all, do no harm.

In business environments the quantitative modeling function needs to be joined at the hip with business activities and processes. Too often they are separate. A popular – and all too often true – image of data-oriented functions in an enterprise setting is that modelers and other “white coat” professionals retreat into the austere confines of their offices, labs and data centers to work their magic, presently emerging with their creations to grandly bestow on the business users. Actual input from these users is a secondary concern, if in fact it is sought at all.

This input is especially critical in dynamic environments like sales & marketing where causal associations can be hard to isolate and quantify. Modeling the variables at play in a complex consumer market is a dauntingly challenging task. They require algorithms that can work in nonlinear relationships, make intelligent insights from incomplete information, and adapt to new input from the field on a real-time basis. And they must reside on a powerful computational platform that can deliver the results to business users on a timely basis for making informed decisions.

At Sentrana, Ali Mahani and my other quantitative modeling colleagues go on “ride-alongs” with our clients – literally accompanying them as they go about their daily business activities. In this way the people who will be designing mathematical formulations related to a client’s business environment have the chance to actually visualize that environment in a practical sense. They can see the context in which decisions are made and better understand how the model’s quantitative insights can help inform the decision process (and, just as importantly, how they can be harmful). They get a better sense for the information decision makers do not have, the time frame in which information needs to be made available to be useful, and the many unpredictable events that can arise and alter the landscape.

Having this vivid mental picture of the actual business environment is helpful for a very important aspect of effective modeling practices – experimentation. Experimentation is a bottom-up process where business professionals provide input that modelers can then build into their assumptions and constraints and test in iterative scenario analyses. Through experimentation, interdisciplinary teams of customer-facing users, business solutions architects and modeling analysts can test the robustness of their assumptions against simulated real world conditions, making necessary adjustments to improve performance. Decision makers will thus go into the field with a higher level of confidence that the model will be able to provide accurate and context-appropriate guidance and recommendations.

All this is not to say that models should not be elegant – or that modelers should not take deserved pride in the beauty of their creations. But their practical goal should be to make the model as simple as possible without shying away from the complexities inherent in the environment it is trying to represent. All models have to simplify to some extent – in practice no simulated system can perfectly capture all the interacting variables and emergent properties of a real environment, especially one as dynamic and variegated as a modern consumer market. But robust business-focused models that incorporate practical bottom-up business insights alongside mathematical abstractions, in an environment of continual testing and recalibrating, have the best chance of ensuring that their contribution to the analysis and decision process will not only do no harm but actually do good.

kinetic energy, not supply & demand

Eighteen years later the answer from many observation posts along the byways of public discourse appears to be: yes, they most certainly do, and so do their fellow travelers, business and financial markets experts.  After all, we just barely survived the most devastating economic event of our times, deeper and more far-reaching than any downturn since the Great Depression, and all the high priests of the field can do is shake their heads and say “wow, I sure didn’t see that coming.”  Distrust of fancy math is rampant in all walks of business life.  That presents a real problem for enterprise decision-makers at a time when they need smart quantitative tools – yes, fancy math and all – more than ever.  Markets are more complex than at any time in human history.  Giant waves of transactional data inundate marketing managers with new information every day.  Managers need science to help them gain valuable insights into the markets for their products and services – but how do they know that the growing number and variety of scientific marketing tools out there aren’t infected with the nasty symptoms of physics envy?

It’s a good question, and one that any decision-maker should ask before embarking on a quantitative marketing solution.  Here are three important questions the manager should ask of any solution being offered:

(a) Is the solution inextricably wedded to a single model of how the world works? This was one of the really fatal flaws in the thinking and modeling practices of economists and financiers in the lead-up to 2008, a flaw most poignantly confessed to by the highest of the high priests of rational economics, Alan Greenspan, in his post-crash testimony to Congress.  Models are supposed to simplify the real world, but not to the point of completely misinterpreting and distorting the behaviors and practices that actually prevail in the world.  Robust quantitative models need to be flexible, adaptable and agnostic in regard to any one single theory that, like rational economics, can become more of a rigid ideology than an objective attempt to explain how the world works.

(b) Is the solution measuring the right thing? Here is where even marketing models with no ideological baggage and with the best of intentions can fall into a trap.  For the past thirty-odd years marketers have tried to define their demand environments through approaches like customer segmentation – identifying demographic segments and then marketing and pricing to the perceived “average” customer in that segment.  A similar approach is segmentation by geography, otherwise known as the “country strategy”.  “How can we optimize our profits in country X?” goes a common problem definition.  But what if the model’s independent variables are (as is often the case) limited to country and product line – answering the question above with a formulation like “sell more turbo widgets in Country X to optimize profits” when in fact the most important influencing variable is actually something else, say a macroeconomic variable like growth in per-capita GDP?  In a world where products proliferate and sales cycles become ever shorter there are numerous variables that influence demand (and hence profitability), and decision-makers need to know these variables at a very granular level – ideally at the level of each potential interaction of customer and product.

(c) Are the right tools being used to measure the right thing? A third pitfall on the road to scientific marketing excellence is the danger of using the wrong tools from the toolkit.  There is a propensity among technology vendors to say things like “our algorithm is better than our competitors’ algorithms”.  In truth there is no one right algorithm because there is no one-size-fits-all solution to anything as complex as markets where product-customer combinations number in the tens of billions.  In this environment there is no such thing as an “average” customer and there is no one single scientific formulation that will solve the problem of making decisions that optimize firmwide performance goals like profitability or market share.  Solutions need to be customized to reflect the unique and constantly evolving contours of each enterprise’s market for its goods and services.

There may be no one perfect, fail-proof screen to detect and avoid the lurking ill effects of physics envy in the market for quantitative business solutions.  But answering a few simple questions like the ones suggested here may go a long way to helping decision-makers avoid the dangers of mathematics for its own sake – and appreciate the value of what mathematical methods can do when applied in the right way for the right reasons.

Looking at the human mind this way, as a machine constantly evaluating possible outcomes based on prior knowledge and assigning probability weights to those outcomes, gave rise to the notion of the “Bayesian brain,” a term popularized in 1983 by Geoffrey Hinton of the University of Toronto and Terry Sejnowski of Johns Hopkins University.  This notion has subsequently received a good amount of validation by neuroscientists as they continue to make advances in understanding how the brain really works.  Neuroscientists Alexandre Pouget and David Knill of the University of Rochester in 2004 referred to a “growing body of evidence that human perceptual computations are ‘Bayes optimal’ (“The Bayesian brain: the role of uncertainty in neural coding and computation,” Trends in Neurosciences, vol 27 issue 12, December 2004, pp 712-719).  That’s a fancy way of saying that we make estimations based on likelihood – for example we routinely make estimations about things like the distance from us of an object, or the speed at which it is traveling, based on our prior knowledge of the shape, clarity and movement patterns of such objects and the likelihood that the present reality fits into that a priori knowledge.

So far, so good.  But there is an Achilles heel to our hierarchical mental gymnastics.  Briefly, we may be great at the kind of proposition-within-proposition reasoning that our sales rep exhibited in getting to the essence of the competitive threat posed by the wholesaler’s move to town.  But we humans are generally pretty lame when it comes to computation.  Our intuitive reasoning skills fail at the task of instantaneously calculating that 34 x 57 X 71 = 137,598.  One way that we get around this failing is through heuristics – basically, shortcuts that we use in conditions of uncertainty to help us get from information to evaluation and decision.  Take the example above of evaluating the distance a particular object might be from us.  Now, there is a physics formula we could apply to measuring distance, height, momentum (if moving) and so forth to give us the precise answer as to how far away that bicyclist in the yellow jersey is from us.  But our brains lack the ability to do spontaneous physics equations.  Even if we precisely knew one or more of the variables it would be hard to do an on-the-spot computation.  So we need something else – a proxy, a heuristic.  That something might be clarity.  Can we see the bicyclist clearly?  Can we make out the details of his black helmet with red stripes, and the ‘Elf Aquitaine’ logo on the yellow jersey?  That can give us enough information to call upon our “cyclist in a yellow jersey” neuronal connections and infer a likelihood that he is, say, 50 yards away.

The problem with heuristics is that they are subject to error: not occasional lapses in judgment but systematic, predictable biases.  For example if we use clarity as a heuristic we may overestimate the distance of that cyclist from us if there is poor visibility.  Understanding the role of heuristic errors in human judgment and decision making was one of the main contributions of Amos Tversky and Daniel Kahneman to our understanding of behavioral factors in human decision-making (“Judgment Under Uncertainty: Heuristics and Biases,” Science New Series vol 185 no. 4157 Sep 1974 pp 1124-1131).  Tversky and Kahneman documented specific heuristic errors such as representativeness (drawing broad or sweeping conclusions from a limited data set), availability (assigning the likelihood of an event based on the easiest example that comes to mind, whether or not appropriate, and anchoring (relying heavily on one piece of information when making a decision even if it is irrelevant).

Since heuristic errors are part and parcel of human judgment and decision-making under uncertainty, we have to take this reality into account when we attempt to integrate quantitative modeling methods and qualitative human judgment.  What are the best mathematical tools and frameworks to integrate these two domains?  One area in which we at Sentrana are spending considerable time is that of Bayesian hierarchical modeling. The Bayesian approach is particularly useful in marketing situations such as modeling differences in the needs and wants of customers using both generalized and conditional assumptions involving multiple variables. Bayesian frameworks provide a natural way to pool disparate sources of information. A Bayesian model requires the formulation of prior distributions and the estimation of a likelihood function, which can add complexity to the model-building process.  However we expect future insights and innovations in this area, alongside the development of more robust computation capabilities to bring more firepower to bear on this difficult but potentially valuable quantitative approach.

It’s hard enough for enterprises to locate those rare possessors of this sales magic and retain their services, but harder still to deal with the fact that in today’s choice-rich, multifaceted demand environments even those talents alone are not sufficient to achieve sales excellence.  We live in a world, after all, where there are purportedly more SKUs (stock-keeping units) on the planet than there are species of living organisms (see for example Eric Beinhocker’s excellent book “The Origin of Wealth”).  A sales representative working for a company with over 100,000 SKUs, which is the norm for large companies in fast-moving goods industries, has to deal with a dimension to the art of the deal that unfortunately has little to do with charm, wits or good grooming: he or she has to figure out on a daily basis which subset of five or six products, out of that universe of tens of thousands, to offer to customers at whatever combination of price points might stand the greatest probability of winning the business.  The computational dimensions of that notion are staggering – quite simply, they are beyond the realm of the feasible when contemplated by the unaided human brain.

Enter technology and the computational powers of quantitative methods.  That which overwhelms the human mind amounts to a few split microseconds of run time for robust data management platforms.  Revenue optimization models can sift through billions of customer-product combinations to recommend pricing configurations with relatively high probabilities of success.  Perhaps these quantitative models could replace those hard-to-find sales skills – after all, if these models can really crunch all that data and recommend prices with the highest likelihood of success, then anyone holding a BlackBerry can access the information and make the sale, right?  Not so fast.  The world may have changed a great deal from 1964, when Mayer and Greenberg produced their article, but intuition is still intuition, and it is no less a necessary ingredient for sales success today than in years past.  For all that computers can achieve, intuition and empathy are simply not things they do.

But is it possible to teach intuition?  At first blush that would seem to be a stretch.  In the minds of many the concept of quantitative methods is intertwined with that of an opaque, algorithm-powered monolith that spits out Delphic recommendations based on historical data crunched through a process unknowable and unviewable by mere mortals – what is commonly (though not always accurately) referred to as a “black box.”  The problem is that in dynamic environments like consumer goods demand markets, decision makers have to negotiate offers based on a kaleidoscope of real-time inputs that require intuitive judgment.  For example, say that you are a distributor in the food services industry and you see a news item that a national wholesaler has opened a discount distribution center in your sales territory.  How would a salesperson process and assign a value to this information?  As human beings, we are uniquely able to compose propositions out of discrete units of information and then embed those propositions within other propositions and so on, creating a hierarchical tree of a limitless number of propositions.

For example, upon reading the headline “National Wholesaler Opens Discount Distribution Center” a sales rep might begin to formulate a succession of hierarchical propositions in rapid sequence:

• wholesaler opens discount distribution center
• wholesaler who is our competitor opens discount distribution center
• wholesaler who is our competitor opens discount distribution center right down the street from our biggest client
• wholesaler who is our competitor and offers everyday low prices opens discount distribution center right down the street from our biggest client
• wholesaler who is our competitor and offers everyday low prices opens discount distribution center right down the street from our biggest client who was a tough price negotiator in our last sale

Our empathetic, capable sales rep will immediately assign a value of high importance to this information and use it to gauge the tone, tenor and negotiating position of the upcoming sales call with this client.  What if the sales rep could also “inform” the quantitative revenue optimization system about this development and have it factored into the ensuing price recommendations ahead of the sales call?

In fact that is possible in today’s environment.  Market awareness models are able to take qualitative human insights, like our sales rep’s awareness of the real-time implications of the competitive threat, and translate them into quantitative factors the models can employ, in conjunction with all the other relevant variables, to produce improved decision support recommendations.  Of course this is not a brainlessly simple exercise: we still face the challenge of translating the sales rep’s instinctual thought process into a language the machine will understand and recognize.  Nonetheless, market awareness models are one of the leading areas of development in the application of quantitative methods to marketing and sales problems.  It all comes back to that basic question posed by Mayer and Greenberg more than 40 years ago: what makes a great salesperson, and how can we best capture and deploy those skills throughout our organization?

But let’s take a different look at this apparent tight coupling of mathematics and dire outcomes. Our recent correspondence with an author who has been widely published on the subject of Wall Street’s use of mathematical models recently offered to us an interesting opinion. His point was that the problem with the models was not so much their complexity, but rather that they were models in the first place. His argument was that you can’t ever perfectly hedge model risk.  Now, I agree with that observation: a model by definition selects some aspects of reality to represent and omits others, and the choice of what to include and what to omit is subject to human error, therefore fallible and not perfectly hedgable.  But I take issue with the idea that the fault lies in the existence of the models themselves.  Models can be misused – I think that much is clear. But the notion that models are all doomed to failure obscures a deeper truth about the goals of predictive modeling; namely that you can seek either to reduce the world or truly explain it. By trying to elegantly reduce the world to as few predictor variables as possible, you are more likely to be sowing the seeds of future failure, because complexity and actual drivers of outcomes are taken out of the equations to make them more solvable (or perhaps sellable, as in the case of the Gaussian copula function that was behind Wall Street’s demise, as we discussed in a previous posting “You Can’t Punt Away the Dimensionality Curse”). Predictive modelers don’t have to go down that road, however: they can also set out with the goal not of reducing an entire system to a single neat, tractable equation, but to quantify and explain all of the relationships that dictate outcomes to the absolute fullest extent possible. Tractability and computability are things to address later in the process, through technological means, but they should not dictate the fundamental mathematical approach at the outset.

As I see it, the problem with the financial market meltdown is not that David Li published an article in the Journal of Fixed Income Securities on the Gaussian copula function, or even that in his article Li, then an analyst with JPMorganChase, identified the price of credit default swap (CDS) contracts as a seemingly elegant proxy for the mortgage market – a proxy that greatly reduced the immense complexity of modeling values and risks in this market but, as it turned out, lost a great deal of critically important information along the way.  No – the real problem was with the incremental decisions practitioners made to adopt this model wholesale, to leverage it up to 50 or more times the worth of the underlying assets, and ultimately to heedlessly employ it as a path to untold riches.  In other words it was the people who used the model, not the model itself.  It was the rating agencies who, in conferring the AAA ratings without which the securities would have never been as widely distributed as they were, assumed that housing prices would never go down.  It was the investment bankers who successfully shouted down the warnings of their internal credit risk departments so that they could sell ever higher volumes of CDOs, with ever-higher levels of leverage, in order to maximize their year-end bonuses.

You can’t fully hedge model risk – that is true.  But you can mitigate model risk through the application of robust decision-making processes.  A model did not take down Wall Street. Models do not “screw up” – they do exactly what they are supposed to do once they have their inputs. The screw-ups occur solely in our application of models to inappropriate situations or to situations which we do not fully understand. The predictions may not reflect reality outcomes as precisely as we wish, but that possibility of error needs to be accounted for by the ultimate decision makers. The output of a model should be an input in any decision process, not the entire decision process. For that reason, the emerging generation of predictive technology solutions being employed by all varieties of business seeks to marry model holism (by including ALL of the relevant variables, rather than the most computationally feasible), computational firepower, and above all, ranges of probability-based recommendations, rather than a single output. It may sound heretical at the moment given the present economic calamity, but as the world gets ever more complex, models will become more valuable to decision makers, not less. Informed, prudent decision-making in regard to those models will not be a luxury, but an absolute necessity.

In trying to answer this postulation we must go back to examine the eternal challenge of good modeling: how to create a simplified representation of reality that in ignoring many real-world features still manages to convey an inherently robust facsimile of the real thing.  For example, one of those maps of New England you buy at Exxon gas stations can serve as a good model for getting you from Hartford, CT to Boston, MA even if it ignores most of the streets and alleyways and other real-world detail that exist along the route.  In their book “Complex Adaptive Systems: An Introduction to Computational Models of Social Life” John H. Miller and Scott E. Page observe that the “ability to ignore is a crucial component of scientific progress”, using the image of a parent’s being able to respond to the incessant “why” questions of a three year old child by saying “just because”.  The trick, as the authors point out, is knowing when (and perhaps more importantly when not) to say “just because”.

While I wholeheartedly agree with that assertion I don’t think that it quite gets us to an adequate level of comfort in applying foresight to the creation of good models.  In his fascinating book “The Origin of Wealth” Eric D. Beinhocker points out that economic modeling took what many consider to be a wrong turn back in the latter years of the 19th century when leading thinkers of the day like Leon Walras and William Stanley Jevons borrowed heavily from the referential context of physics to create models for explaining economic activity, including such notable concepts as a mathematically representable state of equilibrium that continue to serve as the conceptual foundations of modern economics textbooks.  As Beinhocker elaborates, the problem with these models was that some of their fundamental assumptions – like the perfect, robot-like rationality of human beings in making economic choices – didn’t seem to simplify reality as much as contradict reality.  Thus we find ourselves in the present ruminating over the precise, mathematically elegant language of physics and the complex, evolutionary language of biology and debating whether a choice of the wrong science by the founding fathers of economics back in the 19th century led to the failure of models to adequately explain much of what is going on in the economy today and in particular the string of boom-bust upheavals that have become part and parcel of the last 20-odd years of economic activity.

I still don’t think we are there yet in getting closure on the foresight question, but we may be getting closer.  To tie in the strands of thought presented by Miller & Page and Beinhocker, when we get to those basic defining assumptions, when we decide what the model is going to ignore and what it is going to retain, when we say “just because”, that is the point where we have the chance to either build a robust foundation that can withstand the slings and arrows of a continual barrage of economic X-factors or an elegant, commercially viable sandcastle that gets washed away with the next high tide.  If we are going to apply foresight, this is where we have to apply it.

Not that applying foresight is easy, of course, and that is why at Sentrana we are engaged in a continual process of building better models by going back to the basics, asking seemingly simple questions that have complex answers, and trying to ensure that we get it right when we get to that point where it is time to say “just because”.  In our world of Scientific Micromarket Management saying “just because” at the right time can, for example, open up the vista of a complex demand environment to reveal a unique configuration of product mix opportunities at optimal pricing points.  It’s good that we are seeing a heightened level of public discourse about these issues: models may seem to be abstract notions for abstemious minds, but they have real-world economic consequences.  Getting it right has never been more important.

The fault, dear investor, lies not in the head of AIG’s Financial Products Group or members of the Bear Stearns Investment Committee or any other anthropomorphic entity: rather it was a single mathematical formula that apparently felled the pillars of global finance.  That’s the gist of a recent article in the 3.17 edition of Wired magazine entitled “Recipe for Disaster: The Formula that Killed Wall Street” by Felix Salmon.  The formula, known as a Gaussian copula function (when is the last time that term became a fixture of the public discourse?), purported to solve the mother of all securitization problems: establishing default correlation factors between the many constituents of the pools of mortgages and other credit obligations whose cash flows served as the underpinning for the complex derivative securities known as collateralized debt obligations (CDOs).  Awareness of the potential in this arcane formula helped power the CDO market to some $4.7 trillion in volume over the course of the housing bubble years of this decade.  As the Wired article explains, the explosive commercial viability of this formula can be explained by its use of a simple sleight of hand.  Rather than modeling out the default correlation implications of pools of thousands upon thousands of individual mortgage obligations – an extremely complex undertaking requiring powerful algorithms and massively robust computational processing technology – the CDO market’s Wall Street practitioners used a shortcut that appeared elegant but proved deadly: using the market price of credit default swaps (CDSs) as a proxy for the actual historical data.

What happened in essence was that the CDO market ran up against one of the most challenging of quantitative modeling problems: the dimensionality curse.  This refers to what happens in complex environments where numerous variables interact with each other and all of the resulting combinatorial possibilities influence the economic value.  The addition of an incremental variable to the pool exerts an exponential effect on the number of possible outcomes.  Think of a simple case: if you have a pool of two variables then the number of potential outcomes is four: add a third dimension (variable) to the mix and the potential outcomes expand to nine, and so on.  In an environment like pools of thousands of mortgage obligations or credit card receivables influenced by a bevy of macro- and micro-economic, behavioral, seasonal and other random factors there are literally billions of combinatorial outcomes that could affect the incidence, magnitude  and frequency of default events and hence the price of the CDOs whose economic value derives from those pools.   Getting to the right answers – and doing so with enough speed to satisfy the blistering pace of 24-7 investment markets every day – is a daunting challenge to say the least.  So when Daniel X. Li, a quantitative analyst at JPMorgan Chase, posited the use of CDS prices as a proxy for historical data in a 2000 paper published in the Journal of Fixed Income Securities, the CDO market rejoiced and basically punted away the dimensionality curse by using this shortcut.  The reasoning and the assumptions employed proved to be flawed and the disastrous results are entirely visible to the naked eye in all their graphic detail.

In quantitative methods as in life there are no free lunches.  You can’t simply punt away the dimensionality curse – you have to embrace it and try to achieve mastery over it using all the knowledge and technology tools at your disposal.  At Sentrana we deal with dimensionality curse problems every day – the demand markets for the products and services our clients sell are highly complex environments: tens of thousands of products for thousands of customers in hundreds of locations reachable by any number of marketing vehicles and sales channels.  Modeling these environments is not for the faint-hearted: but the problems are not impossible.  The computational technology does exist, as does the modeling science.  The critical ingredient is the will and determination of those who practice quantitative methods in business to forego the easy outs and stay focused on solving the real problems, however daunting.

Perhaps the field of quantitative methods needs a variation of the medical profession’s Hippocratic Oath: First of all, do no harm.  Clearly the Wall Street experiment egregiously failed that standard.  Let’s hope that the next time some arcane mathematical formula figures into the cultural Zeitgeist it will be for better, not for worse.