Elegance does not always lead to the best outcomes

Elegance in modeling is expressed in the appearance of simplicity – rendering sprawlingly complex interrelationships in the real word into the clarity of precise mathematical formulae. Simplicity and elegance are all well and good, unless in the quest for this holy grail you wind up dramatically misrepresenting how things actually work in the environment you are trying to model. This can result in not only failing to solve the business problem at hand, but actually making matters worse than status quo ante by facilitating decisions based on incorrect assumptions. We have a real world example of just how much worse this can be in the financial markets debacle of 2008, when the elegant models crafted by the best and brightest quantitative experts Wall Street had to offer proved to be fatally flawed in the assumptions and heuristics they used to express the variables affecting housing prices, interest rates and mortgage payment trends. Perhaps modelers need to live by something like the Hippocratic oath taken by medical doctors: first of all, do no harm.

In business environments the quantitative modeling function needs to be joined at the hip with business activities and processes. Too often they are separate. A popular – and all too often true – image of data-oriented functions in an enterprise setting is that modelers and other “white coat” professionals retreat into the austere confines of their offices, labs and data centers to work their magic, presently emerging with their creations to grandly bestow on the business users. Actual input from these users is a secondary concern, if in fact it is sought at all.

This input is especially critical in dynamic environments like sales & marketing where causal associations can be hard to isolate and quantify. Modeling the variables at play in a complex consumer market is a dauntingly challenging task. They require algorithms that can work in nonlinear relationships, make intelligent insights from incomplete information, and adapt to new input from the field on a real-time basis. And they must reside on a powerful computational platform that can deliver the results to business users on a timely basis for making informed decisions.

At Sentrana, Ali Mahani and my other quantitative modeling colleagues go on “ride-alongs” with our clients – literally accompanying them as they go about their daily business activities. In this way the people who will be designing mathematical formulations related to a client’s business environment have the chance to actually visualize that environment in a practical sense. They can see the context in which decisions are made and better understand how the model’s quantitative insights can help inform the decision process (and, just as importantly, how they can be harmful). They get a better sense for the information decision makers do not have, the time frame in which information needs to be made available to be useful, and the many unpredictable events that can arise and alter the landscape.

Having this vivid mental picture of the actual business environment is helpful for a very important aspect of effective modeling practices – experimentation. Experimentation is a bottom-up process where business professionals provide input that modelers can then build into their assumptions and constraints and test in iterative scenario analyses. Through experimentation, interdisciplinary teams of customer-facing users, business solutions architects and modeling analysts can test the robustness of their assumptions against simulated real world conditions, making necessary adjustments to improve performance. Decision makers will thus go into the field with a higher level of confidence that the model will be able to provide accurate and context-appropriate guidance and recommendations.

All this is not to say that models should not be elegant – or that modelers should not take deserved pride in the beauty of their creations. But their practical goal should be to make the model as simple as possible without shying away from the complexities inherent in the environment it is trying to represent. All models have to simplify to some extent – in practice no simulated system can perfectly capture all the interacting variables and emergent properties of a real environment, especially one as dynamic and variegated as a modern consumer market. But robust business-focused models that incorporate practical bottom-up business insights alongside mathematical abstractions, in an environment of continual testing and recalibrating, have the best chance of ensuring that their contribution to the analysis and decision process will not only do no harm but actually do good.

Looking at the human mind this way, as a machine constantly evaluating possible outcomes based on prior knowledge and assigning probability weights to those outcomes, gave rise to the notion of the “Bayesian brain,” a term popularized in 1983 by Geoffrey Hinton of the University of Toronto and Terry Sejnowski of Johns Hopkins University.  This notion has subsequently received a good amount of validation by neuroscientists as they continue to make advances in understanding how the brain really works.  Neuroscientists Alexandre Pouget and David Knill of the University of Rochester in 2004 referred to a “growing body of evidence that human perceptual computations are ‘Bayes optimal’ (“The Bayesian brain: the role of uncertainty in neural coding and computation,” Trends in Neurosciences, vol 27 issue 12, December 2004, pp 712-719).  That’s a fancy way of saying that we make estimations based on likelihood – for example we routinely make estimations about things like the distance from us of an object, or the speed at which it is traveling, based on our prior knowledge of the shape, clarity and movement patterns of such objects and the likelihood that the present reality fits into that a priori knowledge.

So far, so good.  But there is an Achilles heel to our hierarchical mental gymnastics.  Briefly, we may be great at the kind of proposition-within-proposition reasoning that our sales rep exhibited in getting to the essence of the competitive threat posed by the wholesaler’s move to town.  But we humans are generally pretty lame when it comes to computation.  Our intuitive reasoning skills fail at the task of instantaneously calculating that 34 x 57 X 71 = 137,598.  One way that we get around this failing is through heuristics – basically, shortcuts that we use in conditions of uncertainty to help us get from information to evaluation and decision.  Take the example above of evaluating the distance a particular object might be from us.  Now, there is a physics formula we could apply to measuring distance, height, momentum (if moving) and so forth to give us the precise answer as to how far away that bicyclist in the yellow jersey is from us.  But our brains lack the ability to do spontaneous physics equations.  Even if we precisely knew one or more of the variables it would be hard to do an on-the-spot computation.  So we need something else – a proxy, a heuristic.  That something might be clarity.  Can we see the bicyclist clearly?  Can we make out the details of his black helmet with red stripes, and the ‘Elf Aquitaine’ logo on the yellow jersey?  That can give us enough information to call upon our “cyclist in a yellow jersey” neuronal connections and infer a likelihood that he is, say, 50 yards away.

The problem with heuristics is that they are subject to error: not occasional lapses in judgment but systematic, predictable biases.  For example if we use clarity as a heuristic we may overestimate the distance of that cyclist from us if there is poor visibility.  Understanding the role of heuristic errors in human judgment and decision making was one of the main contributions of Amos Tversky and Daniel Kahneman to our understanding of behavioral factors in human decision-making (“Judgment Under Uncertainty: Heuristics and Biases,” Science New Series vol 185 no. 4157 Sep 1974 pp 1124-1131).  Tversky and Kahneman documented specific heuristic errors such as representativeness (drawing broad or sweeping conclusions from a limited data set), availability (assigning the likelihood of an event based on the easiest example that comes to mind, whether or not appropriate, and anchoring (relying heavily on one piece of information when making a decision even if it is irrelevant).

Since heuristic errors are part and parcel of human judgment and decision-making under uncertainty, we have to take this reality into account when we attempt to integrate quantitative modeling methods and qualitative human judgment.  What are the best mathematical tools and frameworks to integrate these two domains?  One area in which we at Sentrana are spending considerable time is that of Bayesian hierarchical modeling. The Bayesian approach is particularly useful in marketing situations such as modeling differences in the needs and wants of customers using both generalized and conditional assumptions involving multiple variables. Bayesian frameworks provide a natural way to pool disparate sources of information. A Bayesian model requires the formulation of prior distributions and the estimation of a likelihood function, which can add complexity to the model-building process.  However we expect future insights and innovations in this area, alongside the development of more robust computation capabilities to bring more firepower to bear on this difficult but potentially valuable quantitative approach.

In trying to answer this postulation we must go back to examine the eternal challenge of good modeling: how to create a simplified representation of reality that in ignoring many real-world features still manages to convey an inherently robust facsimile of the real thing.  For example, one of those maps of New England you buy at Exxon gas stations can serve as a good model for getting you from Hartford, CT to Boston, MA even if it ignores most of the streets and alleyways and other real-world detail that exist along the route.  In their book “Complex Adaptive Systems: An Introduction to Computational Models of Social Life” John H. Miller and Scott E. Page observe that the “ability to ignore is a crucial component of scientific progress”, using the image of a parent’s being able to respond to the incessant “why” questions of a three year old child by saying “just because”.  The trick, as the authors point out, is knowing when (and perhaps more importantly when not) to say “just because”.

While I wholeheartedly agree with that assertion I don’t think that it quite gets us to an adequate level of comfort in applying foresight to the creation of good models.  In his fascinating book “The Origin of Wealth” Eric D. Beinhocker points out that economic modeling took what many consider to be a wrong turn back in the latter years of the 19th century when leading thinkers of the day like Leon Walras and William Stanley Jevons borrowed heavily from the referential context of physics to create models for explaining economic activity, including such notable concepts as a mathematically representable state of equilibrium that continue to serve as the conceptual foundations of modern economics textbooks.  As Beinhocker elaborates, the problem with these models was that some of their fundamental assumptions – like the perfect, robot-like rationality of human beings in making economic choices – didn’t seem to simplify reality as much as contradict reality.  Thus we find ourselves in the present ruminating over the precise, mathematically elegant language of physics and the complex, evolutionary language of biology and debating whether a choice of the wrong science by the founding fathers of economics back in the 19th century led to the failure of models to adequately explain much of what is going on in the economy today and in particular the string of boom-bust upheavals that have become part and parcel of the last 20-odd years of economic activity.

I still don’t think we are there yet in getting closure on the foresight question, but we may be getting closer.  To tie in the strands of thought presented by Miller & Page and Beinhocker, when we get to those basic defining assumptions, when we decide what the model is going to ignore and what it is going to retain, when we say “just because”, that is the point where we have the chance to either build a robust foundation that can withstand the slings and arrows of a continual barrage of economic X-factors or an elegant, commercially viable sandcastle that gets washed away with the next high tide.  If we are going to apply foresight, this is where we have to apply it.

Not that applying foresight is easy, of course, and that is why at Sentrana we are engaged in a continual process of building better models by going back to the basics, asking seemingly simple questions that have complex answers, and trying to ensure that we get it right when we get to that point where it is time to say “just because”.  In our world of Scientific Micromarket Management saying “just because” at the right time can, for example, open up the vista of a complex demand environment to reveal a unique configuration of product mix opportunities at optimal pricing points.  It’s good that we are seeing a heightened level of public discourse about these issues: models may seem to be abstract notions for abstemious minds, but they have real-world economic consequences.  Getting it right has never been more important.