The current trade spend paradigm also does not work in the best interests of distributors. While they do benefit in the short term from the financial impact of the trade dollars they receive from their suppliers, distributors do not obtain insights from current trade spend practices that could help them more effectively grow demand across products and categories. Of more benefit would be product and assortment education from their suppliers, enabling them to identify tangible ways to tap into new sources of sales growth.

Category management, a standard practice in many retail sectors that is now gaining currency in foodservice, can be a way to attain this knowledge, use it to effectively drive growth for both manufacturers and distributors, and ultimately to phase out the unproductive aspects of the current trade spend paradigm.

Category management in foodservice can offer the following value proposition:

•    A collaborative platform between manufacturers and distributors based on shared investment of time and resources, as opposed to the general mistrust that permeates trade spend relations
•    Combining knowledge about products, customers and assortment from the manufacturers with transaction-specific insights about timing, location and volume from the distributors to create a holistic view of demand at the micromarket level of every product and every customer
•    Advanced technology to facilitate in-depth analysis and predictive recommendations around all key demand levers e.g. pricing, promotions, assortment, and purchase timing

From “Pay to Play” to “Category Captain”

The common view today in foodservice is that trade dollars are for all intents and purposes a “pay to play” ante required for manufacturers to get their products through distribution channels into the establishments of restaurants and other foodservice operators. These trade dollars are the biggest line item expense after COGS on manufacturers’ income statements, and thus comprise a commensurately large revenue item for their distribution partners. While the conventional wisdom may be that the current system is too entrenched to change, the fact is that a well-executed collaborative category management program can be far more effective than traditional trade spend in identifying the best uses of promotional budgets and delivering on them. Here the manufacturer is not simply cutting a check for some loosely-defined trade campaign and hoping for the best, but instead is taking a more active role in educating, guiding and supporting the distribution partner to increase sales. The vehicle through which the manufacturer can take on this active role is that of category manager, also known as category captain.

In taking on the role of category captain the manufacturer is essentially investing its own time and resources into helping the distributor achieve stated performance objectives. Distributors face a significant challenge in improving category performance. They lack the in-house knowledge about products, customers, and assortment that can help match the right products and offers with the right customers. Manufacturers can supply this knowledge along with supporting tools, including engaging and informative product collateral, suggested product uses, recipes and so forth to help meet objectives such as: increasing demand among existing customers; identifying new customers; and improving sales turn with higher velocity products.

Scalable Category Management

When scaled widely across multiple product categories over time, this approach can ultimately prove to be a more sustainable form of revenue than traditional trade spend dollars. One important difference is that the results delivered by an integrated, data-driven category management program can be measured against quantitative performance targets. Unlike trade spend, where unsystematic and non-automated processes make any kind of ROI analysis problematic or outright impossible, category management puts hard numbers into the hands of decision makers and executives so that they can evaluate how effective the initiatives have been.

The advantages of category management do not just accrue to the distributors. Here again this approach offers up compelling advantages as compared to traditional trade spend. Manufacturers will gain more from their investment of resources into category management than they do from their funding of trade activities. They will have access to a downstream view of demand, supported by actual daily transaction data that has traditionally proven elusive. They will have a much better sense of the timing, quantity and logistical details of customer transactions – and they too will be able to quantify this value with performance measurement tools.

In the prevailing economic climate foodservice is likely to experience strong headwinds to achieving sustainable growth and profitability. Trade spend dollars – which account for about 18% of every sales dollar manufacturers generate – are a dead weight on an industry sector that cannot afford such waste. Replacing the current paradigm with a more efficient, data-driven collaboration model such as category management offers a potential path for industry players to improve their own profitability and that of their industry partners.

The Importance of Collaboration

In foodservice, manufacturers and distributors are the logical partners for a collaborative category management venture. Manufacturers possess deep insights into the product attributes that drive demand for specific customer types, and have a strong understanding of how to manage assortment. On the other side, distributors have the benefit of daily transaction data at a very granular level – what quantities of products in the category are being sold to what locations with what frequency. Combining these insights – ideally through a single integrated data management system able to process inputs from multiple sources and generate insights and actionable recommendations to the relevant decision makers – can create a coherent, unified picture of demand that provides a basis for specific assortment, pricing and promotional activities to grow sales.

Reducing the Guess Factor

A traditional SKU rationalization program may analyze aggregate transaction histories for all the SKUs in a category and mark for elimination some subset of those that occupy the so-called “long tail” – products with sparse data records due to infrequent activity. A typical goal in this regard may be to eliminate 20-25% of all SKUs in the category. The problem with this approach is that without an appropriately detailed level of analytical insight, managers are left to guessing what the resulting effects will be on sales. Transaction frequency is only one variable in presenting a composite picture of demand. For example a certain product may transact on an infrequent basis only, but it may also be a popular niche product with attributes highly valued by major customers. What will the sales impact be of not having this niche product available when a major customer wants to add it to his or her market basket? How can decision makers recognize and differentiate between niche products and other long tail denizens that really deserve to be eliminated from the active product line?

A holistic category management solution, driven by advanced predictive science, can supply answers to these questions. By integrating product attribute knowledge possessed by the manufacturer with quantity and purchase timing data known by the distributor, the system can make recommendations about when to stock the low-frequency but desirable niche items with a higher likelihood of coincidence with the customer’s purchase decision. Techniques such as Hierarchical Bayesian modeling help overcome the analytical challenges typically presented by sparse data. Rather than losing all or part of the customer’s market basket for the sake of an incremental SKU reduction – in most likelihood a losing proposition – the result is retaining a satisfied customer.

Focus on Growing Demand

This approach to category management program shifts attention away from simple cost reduction through inventory rationalization and focuses instead on the revenue side of the equation – growing demand in the category. There are two critical requirements for this to be successful. First, the data management platform must be sufficiently granular to provide meaningful insights at the level of every customer and every product (for example as in the long tail analysis described above). Second, the platform must seamlessly transform into a practical tool which sales representatives can use in the field. This is a particularly important requirement. Foodservice sales & marketing representatives as a rule have very little time for incremental effort above and beyond their existing selling and administrative responsibilities. They certainly do not have sufficient time to juggle multiple sales tools offered by multiple manufacturers acting in the role of category manager. The ideal tool is one with which the representatives have existing familiarity (to avoid time-consuming learning curves for new processes) and which can seamlessly integrate data from multiple input sources.

Manufacturers and distributors need more than just a rationalization program to optimize performance at the category level. A holistic approach, supported by robust analytics delivering actionable real-time guidance to sales professionals in the field, can improve category performance all along the foodservice value chain.

These are all valid concerns.  The problem is that the decision makers are sitting in different parts of the organization, their objectives are often in  conflict with each other (or at the very least require trade-offs and compromises), and they are not armed with sufficient information to understand the broader impact of each price decision on firmwide performance.

Coordinating all these disparate pricing activities is a daunting challenge, but a critically necessary one.  Pricing is one of the single most important levers an enterprise operates to influence performance.  Various research studies have shown that a 1% increase in the average price of goods and services for a typical Global 1200 company can lift operating profits by more than 8.5%.  With stakes like these, decision makers cannot simply kick the can down the road and hope for the best.  So how do you solve your organization’s coordination challenge and align pricing activities for optimal performance?

Let’s break the coordination challenge into three component parts: where decision makers are sitting, what data they have access to, and how they are making decisions.

organizational silos can impede effective decision making

Most likely they are sitting in organizational silos.  The term “silo” is used to describe discrete islands of activities within the enterprise: each one self-contained and disconnected from the others. The term carries a negative connotation; yet it is in many ways a natural, probably necessary byproduct of the increased complexity of the business landscape.  Over the past twenty years the number of products, customer segments, geographic regions and sales channels enterprises have to deal with has exploded.  The activities that businesses have to perform to effectively serve their markets have multiplied in number and are far more specialized – hence the silo.

The challenge is not how to make the silo go away, but how to overcome the silo mentality.   Here is where we introduce the “what” factor: in addition to where pricing decision makers are sitting in the organization, we need to understand what data they have access to for making decisions.  What we are looking for is a way to ensure that people in different silos have access to the same information – that they have a transparent view into the demand environment from which to make informed, coordinated pricing decisions.

That is easier said than done.  Firmwide enterprise resource planning (ERP) systems can be an important first step by having a system of record and system of execution to provide integrated data visibility throughout the organization.  That by itself is a tough enough challenge – ERP system integration is a notoriously complicated, expensive, resource-consuming process.  But it is still only a partial step to achieving pricing coordination.  It addresses the issue of what data pricing decision makers can access and analyze, but not necessarily how they are using the data:

• How can a salesperson in the field obtain precise guidance about price recommendations for a specific customer-product combination?

• How could that same salesperson get additional insight about opportunities for cross-sell to increase share of basket?

• How can a pricing analyst at corporate headquarters create new pricing rules without a cumbersome internal process requiring support from the IT department?

• How can her manager respond in real time to an exception request from the field?

• Finally, how can all these activities happen seamlessly in a timely manner without being at cross-purposes with each other?

Answering the “how” question requires specialized tools that can perform the activities particular to each silo where pricing decisions are being made, while at the same time being able to connect into the system of record to make available the data necessary for common visibility.  For companies with thousands of customers and products you need the leverage of advanced science and powerful computational technology to make sense of the environment, better understand the behavior of your customers, and provide intelligent, informed recommendations armed with knowledge about demand signals, price rules, cost-to-serve, and the other critical factors that influence price.

This coordination challenge is not easy, even with the right decision support tools.  But remember the stakes – matching the right customer with the right product at the right price can have a profound effect on firmwide performance measures such as market share and profitability.    It’s a challenge you can ill afford not to solve.

A mere three dimensions of complexity

Remember the Rubik’s Cube?  That delightfully maddening cultural relic of the 1980s was challenging because you had to configure the right sequence of moves in three dimensions.  One small misstep – one rotation along the wrong axis – and the whole strategy would fall apart.  Now, think of trying to solve a Rubik’s Cube-like puzzle, not in three dimensions but in at least five!  Our visual cortex regions boggle trying to imagine what this hypercube would even look like.  Yet that is the gauntlet thrown down to campaign marketing managers: configure the (1) right customer with the (2) right product and the (3) right promotional offer using the (4) message via the (5) right channel.  A typical challenge of this nature presented itself to one of our clients recently: configure eight potential messages to 50 customer segments in 70 regional markets concerning 50 product categories and four distribution channels:

8 x 50 x 70 x 50 x 4 = 5.6 million unique campaigns   for budget consideration!

That is obviously a larger number of alternative spending choices than the unaided human brain can reasonably analyze.  But the complexity doesn’t end there.  With the old Rubik’s Cube there was only one objective: get each face of the cube to be one single color.  Not so with the Marketing Hypercube (pictured in the diagram).  There are in fact multiple potential target outcomes of any given campaign.  Is the objective to build initial awareness of the company or the product?  Or is it to instill preference among an audience already familiar with the product?  Or, alternatively, is it to maximize actual purchases through targeted prices, promotional incentives, penetration opportunities, and/or purchase timing strategies?  In effect, the targeted sales & marketing outcomes themselves represent yet another dimension of complexity.

Multi-dimensional campaign marketing challenges

So how do we solve a problem of this magnitude of complexity?

Perhaps it is somewhat counterintuitive, given that we have called for a strong dose of scientific rigor, but the first order of business is to put aside the mathematics, take a step back and employ some good old-fashioned human judgment (don’t worry, we’ll shortly come back to the mathematics when we start to build predictive models around customers, messages and objectives).  Let’s start by remembering what we are trying to accomplish: to configure a campaign that will most effectively resonate with the target customer segments and accomplish our specified performance objectives.   We want to be able to predict the effect of the campaign before it is even launched.  This requires making some basic assumptions – but before your analysts integrate these assumptions into predictive models they need to obtain bottom-up business insights. These insights come from experience gained by your sales associates through interaction with their customers.  For example, they can be gleaned from short 30-45 second surveys and similar diagnostic tools built around particular initiatives (e.g. price, penetration, wallet share, loyalty, and general awareness-familiarity-preference survey templates).

The next challenge is to align these insights with the right segments.  Don’t think of this as a “once-and-done” event.  You have hundreds of thousands of customers and there are near-limitless ways to segment them.  The segments around which you build your first campaign iteration may not be the segments you employ in the end – or perhaps you will learn that those segments require different campaign strategies.  This is an iterative process – sampling, inputting new insights into existing predictive models, aligning campaigns to segments, resampling, revising segment strategies, updating model assumptions and constraints, and repeating.

It may sound tedious.  But over time this iterative process will help you greatly improve the accuracy of your predictive campaign models.  You will be in the position to pinpoint the effects that a specific campaign had on improving the value of certain customers’ transaction baskets through penetration initiatives, for example, or to measure the contribution of a customer loyalty campaign to actual revenue saved through decreased contract defections.  Those 5.6 million alternative budget allocations will start to look less daunting, and you will have a higher degree of confidence in making spending decisions closely aligned at a very granular level with your demand environment (for example, our client was able to more than triple its customer conversion rate through applying science to its campaign marketing process).  In short, you will be able to do more with less – even if you are one of the many people who never did get the hang of the Rubik’s Cube!

We live in the Age of Information, so we are told.  Never before has so much raw data existed bearing testament to every  pulsebeat of human commerce, every touchpoint between a customer and a good or service.  The problem for decision-makers, according to the conventional wisdom, is Information Overload – volumes more data to analyze than the human brain can easily digest.  But it is not that simple – there are deeper challenges below the surface.

Information is not always where you need it

While the conventional wisdom is right in the aggregate, the lush and dense information rainforest starts to turn remarkably arid and sparse as you drill down into the nuanced segments of your demand environment.  At the micromarket level, infrequent transactional activity in the long tail of customers and SKUs yields little insight to inform decision making.  Managers thus face challenges that go well beyond the simplistic construct of TMI (too much information).  They need tools for managing the real information problems in their micromarkets.  These tools need to address head-on the challenges posed by what we call the 4-Cs:

• Complexity: With near-limitless combinations – of customers, products, locations, messages and channels – managers need the ability to first aggregate and then disentangle how variables related to price, assortment, advertising & promotions, and sales mechanisms affect customer demand and thus impact firmwide performance metrics like market share or profit margin.  Not knowing what impacts sales or profits raises the risk of suboptimal performance. Advanced scientific methods can help fill in the gaps where data sparsity exists and extend the vision of key decision-makers far into the details of what moves their markets.

• Coordination: Marketing involves a series of decisions, all of which have an impact on each other – yet each decision often gets made in an organizational silo isolated from other decisions.  This can produce persistently suboptimal outcomes unless managers can overcome the limitations of organizational silos.  Holistic optimization tools that provide visibility across silos and facilitate “what-if” experimentation can help achieve a clear, coordinated understanding of each single decision in a more integrated context.

• Connection: Managers need to connect to what the market is telling them in real time.  Historical transaction data can only help so much in an environment of constant flux: customer tastes change and competitive threats emerge in a Petri dish of constantly evolving activity.  It is not enough for decision-makers to learn from their quantitative systems: the systems have to learn from them as well.  This is what it means to be market aware: intimately connecting human experience and judgment with machine-based algorithms for optimal decision guidance.

• Customization: Insightful managers know that there is no such thing as an “average” customer.  Marketing and sales messages that play to a perceived average will wind up being average themselves – in other words falling short of truly connecting in the best way with target customers.  The fact is that every enterprise’s customer base is unique – defined by a distinctive combination of tastes, wants, needs and propensity to spend.  This is true even if the product line is what most observers would view as commodities.  Customizing a value proposition down to the most granular level possible can unlock the power of micromarket monopoly and defend against the margin-eroding practices of cutthroat price competition.

If the Information Age were really all about combing through volumes of aggregate data to develop key marketing decisions for your average customers then the 4-C framework would not matter so much – you could price, advertise and sell based on their perceived wants, needs and spending propensities.  But that average customer doesn’t exist.  The more precisely you can gain the necessary insights into micromarket uniqueness, the more you can calibrate marketing and sales decisions to optimal advantage.

So, if you are a marketing manager in a highly competitive industry  like foodservices, consumer packaged goods or retail, what should you be looking for in business intelligence & analytical solutions to take on the 4-C challenges?  Ask yourself three questions:

1. Can the solution really cope with the complexity of my demand environment in a way that is commercially viable, i.e. that keeps up with the fast pace of my daily decision-making?
2. Is the solution seamlessly compatible with my company’s existing technology platform including existing ERP and other critically important business intelligence?
3. Can my sales reps continue to do their jobs effectively and impart their experience and judgment without compromising the integrity of the system’s recommendations?

We’re going to come back and explore each of these questions in subsequent postings.

Olay, Nivea, Neutrogena and L’Oreal are all established beauty products brands with a broad array of medium-priced product lines and multiple product offerings in each.  More to the point, for purposes of this thought exercise of mine, is that each of them offers a range of good quality facial cleansers, a product I buy on average about once every two months.  The exercise was to determine what, if any, brand loyalty existed in my facial cleanser purchases over the last 2 years.  The answer appeared to be: none.  Nada.  At some point over those past 24 months and (give or take) 12 purchases, my domestic shelf space has been occupied by at least one representative facial cleanser SKU from each of those brands.  I wondered why this was the case.  And then I remembered that it was not always thus.  Long ago (more years than I care to disclose) there was a rather splendid product by Neutrogena called the Facial Cleansing Bar.

a simpler time for consumers

It was a large amber, translucent bar of pure goodness, I thought at the time, and for many years it was the only thing that would ever come to mind in association with facial cleansing. That product still exists, but the last time I bought a bar was back in an era when folks were marveling over the newfound wonders of that thing called email… I wondered: what had happened along this journey from the devoted, faithful me of old to this fickle consumer circa 2010?  What could any one of these companies do to win back my loyalty, and presumably that of many others like me?

Brand success depends on both walletshare and mindshare. If a brand manager wants to get to my wallet then he or she has to first get to my mind and convince me why, out of all the marketing messages that assault me with mind-numbing regularity throughout the changing venues and vistas of my daily routine, this is the one that is most worthy of my time and money.  The problem is that our economy is awash in multiple products, multiple messaging formats and multiple physical & digital marketing channels.  More brands than ever before vie for our attention and our dollars (or rupees, or renminbi); and in so doing the ability of any given brand to make a meaningful impact is diluted by the sheer magnitude and frequency of audio-visual-textual images clamoring to engage our senses.  This, it seems, is what happened to that wonderful amber cleansing bar by Neutrogena – it got lost in the proliferation of categories, products and beauty care messages that exploded into our lives over the last 20-odd years.  This proliferation explosion has engendered many results both positive and negative – one of the latter of which has been to weaken the brand’s ability to connect heart and mind.

If mindshare leads to walletshare, then heartshare leads to mindshare. I don’t believe this paradigm has changed – I think it is no less true today than it was in the golden age of brand advertising (see any episode of Mad Men for a useful reference point).  But the path to the heart is different today, and much trickier.  It’s not all about the brilliant ad that manages to imprint the differentiating qualities of its product so firmly in the cultural mindset that whole households could recite them in their sleep (it is still partly about that, but much less so than the days when the Don Drapers of the world dreamed up the copy over three-martini lunches).  In fact it is not about any one thing, but rather about multiple things – things that form a complex multi-dimensional mathematical equation that would have those ad men of old reaching for the Scotch bottles they kept in their office credenzas.  Namely: how do you match the right message with the right format, for the right geographic region and customer segment via the right marketing channel, for optimal effect? The answer to this equation is not obvious – in fact it is entirely unknowable to the unaided human brain.  To solve it requires vast computational power and advanced scientific methods for solving multivariable problems where many of the variables are interdependent (which adds several levels of complexity to the math needed to get to the solution).

To put it more mundanely, it requires disentangling the clues from that vast sea of transactional information that form a composite picture of the customer-product interaction leading to my walking to the CVS checkout counter with a particular facial cleanser in hand.  What are the demand chain activities that could make this customer-product interaction more predictable; and more personally satisfying for me, the customer?  Is it a pricing question, or a product mix question, or a channel promotions question or an image question?  Traditionally, marketing managers have viewed these as separate issues rather than forming an integrated portfolio of activities around which to optimize a particular objective (like brand value).  But they are not separate, and they cannot be optimally solved in the isolation tanks of marketing department silos.

Fortunately for the beleaguered brand decision-makers of the world there are holistic quantitative marketing solutions that can help them leverage the insights necessary to build and sustain their brand’s value by treating these questions as components of an integrated whole.  The starting point for these solutions is the recognition that what happens in one part of the demand chain affects things that happen in other parts.  As a consumer, my world is more complex now than it used to be and most likely my attention span for any one message is shorter.  But some configuration of activities undertaken by a company – or more than one company along a supply chain – can have a significant impact on me at that reinforces the value of the brand and gives it a meaning more closely aligned with how I actually make purchasing decisions today.

These solutions may lack the simple goodness of that old facial cleansing bar – but they may yet win the heartshare of facial cleanser (and many other) consumers all the world over.

But let’s take a different look at this apparent tight coupling of mathematics and dire outcomes. Our recent correspondence with an author who has been widely published on the subject of Wall Street’s use of mathematical models recently offered to us an interesting opinion. His point was that the problem with the models was not so much their complexity, but rather that they were models in the first place. His argument was that you can’t ever perfectly hedge model risk.  Now, I agree with that observation: a model by definition selects some aspects of reality to represent and omits others, and the choice of what to include and what to omit is subject to human error, therefore fallible and not perfectly hedgable.  But I take issue with the idea that the fault lies in the existence of the models themselves.  Models can be misused – I think that much is clear. But the notion that models are all doomed to failure obscures a deeper truth about the goals of predictive modeling; namely that you can seek either to reduce the world or truly explain it. By trying to elegantly reduce the world to as few predictor variables as possible, you are more likely to be sowing the seeds of future failure, because complexity and actual drivers of outcomes are taken out of the equations to make them more solvable (or perhaps sellable, as in the case of the Gaussian copula function that was behind Wall Street’s demise, as we discussed in a previous posting “You Can’t Punt Away the Dimensionality Curse”). Predictive modelers don’t have to go down that road, however: they can also set out with the goal not of reducing an entire system to a single neat, tractable equation, but to quantify and explain all of the relationships that dictate outcomes to the absolute fullest extent possible. Tractability and computability are things to address later in the process, through technological means, but they should not dictate the fundamental mathematical approach at the outset.

As I see it, the problem with the financial market meltdown is not that David Li published an article in the Journal of Fixed Income Securities on the Gaussian copula function, or even that in his article Li, then an analyst with JPMorganChase, identified the price of credit default swap (CDS) contracts as a seemingly elegant proxy for the mortgage market – a proxy that greatly reduced the immense complexity of modeling values and risks in this market but, as it turned out, lost a great deal of critically important information along the way.  No – the real problem was with the incremental decisions practitioners made to adopt this model wholesale, to leverage it up to 50 or more times the worth of the underlying assets, and ultimately to heedlessly employ it as a path to untold riches.  In other words it was the people who used the model, not the model itself.  It was the rating agencies who, in conferring the AAA ratings without which the securities would have never been as widely distributed as they were, assumed that housing prices would never go down.  It was the investment bankers who successfully shouted down the warnings of their internal credit risk departments so that they could sell ever higher volumes of CDOs, with ever-higher levels of leverage, in order to maximize their year-end bonuses.

You can’t fully hedge model risk – that is true.  But you can mitigate model risk through the application of robust decision-making processes.  A model did not take down Wall Street. Models do not “screw up” – they do exactly what they are supposed to do once they have their inputs. The screw-ups occur solely in our application of models to inappropriate situations or to situations which we do not fully understand. The predictions may not reflect reality outcomes as precisely as we wish, but that possibility of error needs to be accounted for by the ultimate decision makers. The output of a model should be an input in any decision process, not the entire decision process. For that reason, the emerging generation of predictive technology solutions being employed by all varieties of business seeks to marry model holism (by including ALL of the relevant variables, rather than the most computationally feasible), computational firepower, and above all, ranges of probability-based recommendations, rather than a single output. It may sound heretical at the moment given the present economic calamity, but as the world gets ever more complex, models will become more valuable to decision makers, not less. Informed, prudent decision-making in regard to those models will not be a luxury, but an absolute necessity.

The fault, dear investor, lies not in the head of AIG’s Financial Products Group or members of the Bear Stearns Investment Committee or any other anthropomorphic entity: rather it was a single mathematical formula that apparently felled the pillars of global finance.  That’s the gist of a recent article in the 3.17 edition of Wired magazine entitled “Recipe for Disaster: The Formula that Killed Wall Street” by Felix Salmon.  The formula, known as a Gaussian copula function (when is the last time that term became a fixture of the public discourse?), purported to solve the mother of all securitization problems: establishing default correlation factors between the many constituents of the pools of mortgages and other credit obligations whose cash flows served as the underpinning for the complex derivative securities known as collateralized debt obligations (CDOs).  Awareness of the potential in this arcane formula helped power the CDO market to some $4.7 trillion in volume over the course of the housing bubble years of this decade.  As the Wired article explains, the explosive commercial viability of this formula can be explained by its use of a simple sleight of hand.  Rather than modeling out the default correlation implications of pools of thousands upon thousands of individual mortgage obligations – an extremely complex undertaking requiring powerful algorithms and massively robust computational processing technology – the CDO market’s Wall Street practitioners used a shortcut that appeared elegant but proved deadly: using the market price of credit default swaps (CDSs) as a proxy for the actual historical data.

What happened in essence was that the CDO market ran up against one of the most challenging of quantitative modeling problems: the dimensionality curse.  This refers to what happens in complex environments where numerous variables interact with each other and all of the resulting combinatorial possibilities influence the economic value.  The addition of an incremental variable to the pool exerts an exponential effect on the number of possible outcomes.  Think of a simple case: if you have a pool of two variables then the number of potential outcomes is four: add a third dimension (variable) to the mix and the potential outcomes expand to nine, and so on.  In an environment like pools of thousands of mortgage obligations or credit card receivables influenced by a bevy of macro- and micro-economic, behavioral, seasonal and other random factors there are literally billions of combinatorial outcomes that could affect the incidence, magnitude  and frequency of default events and hence the price of the CDOs whose economic value derives from those pools.   Getting to the right answers – and doing so with enough speed to satisfy the blistering pace of 24-7 investment markets every day – is a daunting challenge to say the least.  So when Daniel X. Li, a quantitative analyst at JPMorgan Chase, posited the use of CDS prices as a proxy for historical data in a 2000 paper published in the Journal of Fixed Income Securities, the CDO market rejoiced and basically punted away the dimensionality curse by using this shortcut.  The reasoning and the assumptions employed proved to be flawed and the disastrous results are entirely visible to the naked eye in all their graphic detail.

In quantitative methods as in life there are no free lunches.  You can’t simply punt away the dimensionality curse – you have to embrace it and try to achieve mastery over it using all the knowledge and technology tools at your disposal.  At Sentrana we deal with dimensionality curse problems every day – the demand markets for the products and services our clients sell are highly complex environments: tens of thousands of products for thousands of customers in hundreds of locations reachable by any number of marketing vehicles and sales channels.  Modeling these environments is not for the faint-hearted: but the problems are not impossible.  The computational technology does exist, as does the modeling science.  The critical ingredient is the will and determination of those who practice quantitative methods in business to forego the easy outs and stay focused on solving the real problems, however daunting.

Perhaps the field of quantitative methods needs a variation of the medical profession’s Hippocratic Oath: First of all, do no harm.  Clearly the Wall Street experiment egregiously failed that standard.  Let’s hope that the next time some arcane mathematical formula figures into the cultural Zeitgeist it will be for better, not for worse.