These are all valid concerns.  The problem is that the decision makers are sitting in different parts of the organization, their objectives are often in  conflict with each other (or at the very least require trade-offs and compromises), and they are not armed with sufficient information to understand the broader impact of each price decision on firmwide performance.

Coordinating all these disparate pricing activities is a daunting challenge, but a critically necessary one.  Pricing is one of the single most important levers an enterprise operates to influence performance.  Various research studies have shown that a 1% increase in the average price of goods and services for a typical Global 1200 company can lift operating profits by more than 8.5%.  With stakes like these, decision makers cannot simply kick the can down the road and hope for the best.  So how do you solve your organization’s coordination challenge and align pricing activities for optimal performance?

Let’s break the coordination challenge into three component parts: where decision makers are sitting, what data they have access to, and how they are making decisions.

organizational silos can impede effective decision making

Most likely they are sitting in organizational silos.  The term “silo” is used to describe discrete islands of activities within the enterprise: each one self-contained and disconnected from the others. The term carries a negative connotation; yet it is in many ways a natural, probably necessary byproduct of the increased complexity of the business landscape.  Over the past twenty years the number of products, customer segments, geographic regions and sales channels enterprises have to deal with has exploded.  The activities that businesses have to perform to effectively serve their markets have multiplied in number and are far more specialized – hence the silo.

The challenge is not how to make the silo go away, but how to overcome the silo mentality.   Here is where we introduce the “what” factor: in addition to where pricing decision makers are sitting in the organization, we need to understand what data they have access to for making decisions.  What we are looking for is a way to ensure that people in different silos have access to the same information – that they have a transparent view into the demand environment from which to make informed, coordinated pricing decisions.

That is easier said than done.  Firmwide enterprise resource planning (ERP) systems can be an important first step by having a system of record and system of execution to provide integrated data visibility throughout the organization.  That by itself is a tough enough challenge – ERP system integration is a notoriously complicated, expensive, resource-consuming process.  But it is still only a partial step to achieving pricing coordination.  It addresses the issue of what data pricing decision makers can access and analyze, but not necessarily how they are using the data:

• How can a salesperson in the field obtain precise guidance about price recommendations for a specific customer-product combination?

• How could that same salesperson get additional insight about opportunities for cross-sell to increase share of basket?

• How can a pricing analyst at corporate headquarters create new pricing rules without a cumbersome internal process requiring support from the IT department?

• How can her manager respond in real time to an exception request from the field?

• Finally, how can all these activities happen seamlessly in a timely manner without being at cross-purposes with each other?

Answering the “how” question requires specialized tools that can perform the activities particular to each silo where pricing decisions are being made, while at the same time being able to connect into the system of record to make available the data necessary for common visibility.  For companies with thousands of customers and products you need the leverage of advanced science and powerful computational technology to make sense of the environment, better understand the behavior of your customers, and provide intelligent, informed recommendations armed with knowledge about demand signals, price rules, cost-to-serve, and the other critical factors that influence price.

This coordination challenge is not easy, even with the right decision support tools.  But remember the stakes – matching the right customer with the right product at the right price can have a profound effect on firmwide performance measures such as market share and profitability.    It’s a challenge you can ill afford not to solve.

There is no shortage of quantitative analysis in the trade spend practices of foodservice manufacturers.  Unfortunately, very little of this analysis helps give decision-makers insights about the effectiveness of their trade spend programs.  The numbers being crunched do not relate to signals about actual downstream demand, but rather to the formidable mountain of claims from their distributors.  These claims come in all manner of data formats and accounting entries and it typically takes armies of brokers, salespeople and financial staff to figure them out.  After all the cumbersome and error-prone line-by-line calculations to validate claims are said and done, you are no more informed about the profitability or the potential risks associated with any given program.  No wonder there is widespread dissatisfaction with the effectiveness of these programs.  Over 75% of manufacturers in this sector consider their trade spend initiatives to be inefficient, according to the 2010 MarketIntelligence Foodservice Trade Survey.

Pricing signals matter for getting the most from trade spend activities

Decision-makers at foodservice manufacturers need a new approach: one that creates greater visibility throughout complex information chains; and applies analytical methods in order to better align and optimize trade decisions with pricing and other key marketing levers.  Abundant data exist, as do the analytical methods to gain insights from them.  Better measurement and analysis can lead managers to more profitable decisions for themselves as well as their trade partners. This can help turn trade spend into trade investment.

Low-tech, non-standardized processes generate waste

The hodge-podge of disparate programs scattered around the organization with a variety of process and data formats do not easily lend themselves to effective measurement, performance tracking, or coordination with other key marketing and pricing decisions.  Programs tend to have non-standardized and duplicative contracts, cumbersome claims and dispute resolution procedures, and generally low-tech operational processes.  Manufacturers have little way of knowing whether the dollars they are putting into these programs are having measurable impact at the operator and patron level or whether they are simply staying in the pockets of the distributors.  The complexity of the information chain creates a tremendous amount of waste in the system over time that negatively impacts profitability throughout the chain.

New trends in distributor pricing mean opportunities for manufacturers

Such archaic practices stand in sharp contrast to a sea change taking place in distributor pricing: namely, the growing trend of setting prices according to downstream patron and operator demand rather than based on an arbitrary mark-up on the zero sum negotiated price between manufacturers and distributors.  Scientific pricing, an increasingly prevalent practice in the food services wholesale space, offers predictive demand insights for each potential product and customer combination.  Prices thus contain more information about actual downstream demand, enabling products to be pulled through the channel rather than pushed downstream based on the subjective outcomes of manufacturer-distributor negotiations.  Manufacturers have an opportunity to use the same demand signals that inform scientific pricing to guide a more accurate allocation of their trade funds to drive greater overall volume and profit.

Social networking is now standard operating procedure for many restaurant-goers

Let the Facebook Generation work for you

These demand signals are especially relevant because technology has thoroughly transformed the way that retail operators (such as restaurants and caterers) and their patrons communicate.  Digital social networking is now an established way of life for a rapidly growing group of Americans, the majority of whom fall within the most desirable demographic segments of the consumer market.  Sites like Yelp, Urban Spoon and TripAdvisor ensure that salient details about a given restaurant’s menu, prices, food quality, social environment and numerous other attributes are readily available at the fingertips of smartphone-wielding prospective patrons preparing to decide where to gather and dine for the evening.  Clearly, operators have strong incentives to match demand with available supply.  For manufacturers this means abundant information coming from points downstream that can help inform smart trade promotion and pricing decisions.  Decision-makers can gain insights about demand as it relates to geographic and demographic segments; further refine this understanding as it pertains to product categories; and experiment with alternative what-if scenarios to predict the effect of various trade promotion and pricing decisions on demand.

More about trade spend on Sentrana’s blog

In the coming weeks we will be spending some more time on this blog site looking in detail at different aspects of the trade spend challenge and the opportunities we see for foodservice manufacturers to improve performance.  Forthcoming areas of focus include: collaborative campaigns to create win-win programs with trade partners; trade program design; issues related to program execution; and other topics that can help reveal active ways to turn trade spend into trade investment.
?

Multiple optimal prices can exist for a product, even in transparent markets. Note that all of the prices in this image apply to the exact same HP printer.

The lessons of efficiently cooling a home can be applied to many scenarios. In business, having a system in place for tweaking procedures continuously is easier to manage over time than are prolonged periods of stasis followed by dramatic transformations. Transformations are complicated. They are often expensive. If too much time passes between transformations, the organization’s inertia coefficient (a 100% made-up term) passes a critical threshold. After that point, two outcomes are the most likely, with a few shades of gray in between: (1) transformation projects mushroom from merely “expensive” to “expensive and painful”, or (2) the company is too lethargic to change, effectively dooming the business to eventual defeat or absorption by more innovative rivals. For the sake of comprehensiveness, I have to acknowledge that for a fortunate few, “federal bailout” must now be added to this list as a third possible outcome. However, in a few years we will see if my suspicion that outcome three eventually finds its way back to outcome two turns out to be correct.

Unlike management theory and the social sciences, the physical sciences rest on a set of principles which can serve as a bedrock for many different streams of work without having to re-establish the foundation. When we put the rover on Mars, we did not have to re-establish that gravity exists. We did not have to re-create Galileo’s experiments by bringing the town of Pisa’s only distinctive landmark to another planet in order to drop the balls from the window again. Experimentation occurs of course, and advances such as quantum mechanics and relativity are challenged, but the principles used to conduct research remain constant for the most part. The principles of management and competitive strategy have no business being treated the same way, however. For an organization to remain healthy, it must continuously challenge and question its assumptions about how best to manage its employees and connect with its customers. By not doing this, businesses invite future competitive disadvantages as static processes outlive their usefulness and simply become too ingrained in the organization to change.

Pricing is a corporate discipline that is unfortunately in need of transformation at most companies. The problem is that most companies, especially those with extremely large numbers of products, customers or both, are incapable of making consistent changes to their pricing strategy or practices. They don’t have the people, tools or the required knowledge to make these adjustments in a principled way. Yet pricing decisions are among the most impactful ones that a company makes when it comes to top- and bottom-line performance. So what do businesses need in order to be able to make constant tweaks to optimize their prices?

First, you need a model. In fact, many models are required. As I have stated before in this space, each price that a business shows to the market is a bet that the figure on the price tag is the one that will bring the greatest profit to the business. What you’re really betting on is the customer’s response: when they see the price, will they buy or reject? To predict the probability of customer response, you need mathematical models. Without them, you are not betting, you’re gambling. Now, it stands to reason that each customer is a little bit different from the one next to him or her. There are many broad similarities that can be identified among customers such as their age, income, gender, ethnicity and so on, but it is the differences that really determine how much a customer is willing to pay for something. Determining what these differences are and how to adjust prices to account for them requires sophisticated analysis of the conditions surrounding past transactions as well the specific attributes of every product that you sell and every customer that you have. It’s not child’s play, true, but it is critical to pricing in the 21^st Century.

A business with 100,000 SKUs and 400,000 products actually has (100,000 * 400,000) = 40 Billion individual customer-product combinations that can be priced!

To really tweak prices at the most granular level, individual models for every customer-item pairing would be required. Remember, a single model for each customer won’t do the job. Personally, I am a completely different customer when I am in the market for fresh fish than I am for gym socks. I buy fish based on freshness without much regard for price, but I buy socks based on price and price alone. A model that labeled me as either “highly price conscious” or “completely insensitive to price” would incorrectly predict my behavior in most circumstances. Similarly, a model built to predict my behavior that somehow averaged out my different price sensitivity levels by labeling me as “medium price-conscious” would be wrong for both examples above. So the need for customer-product-specific models is clear, but that leads to a second issue. Customers’ preferences change over time, and so does their wiliness-to-pay for each item. Models need to adjust regularly as well, possibly even every day.

Pricing can truly become a source of competitive advantage in business if the organization internalizes the ability to adjust pricing models continuously. Why? Pricing at the micro-market level (as we refer to it at Sentrana) requires an extraordinarily nuanced understanding of your customers. Predicting customer response at the individual product level means that your prices are not leaving any money on the table. Moreover, re-building the models continuously to incorporate the newest data (with some mathematical tricks to ensure that important shifts are not ignored or misinterpreted) gives the business a way to continuously learn what the market wants, and what they will pay for it. For this reason, businesses must acclimate themselves to the fact that pricing is a continuous process of discovery, rather than a periodic exercise.

In my next post, I will explain how such a seemingly daunting idea can be automated and become a functioning part of the enterprise fabric.

To answer this question, let’s take a closer look at the solution itself. Dynamic pricing is a form of yield management (also called targeted pricing, flexible pricing, tailored pricing or discriminatory pricing), which formally emerged in the mid-1980s as a means for airlines to capture some value from plane seats that would otherwise go empty by offering, for example, lower than published fares for customers willing to forego other benefits (such as the ability to change a flight date or cancel the ticket). This breakthrough science allows organizations to understand, anticipate and influence consumer behavior in order to maximize revenue or profits from a fixed, perishable resource (e.g. airline seats, hotel room reservations, etc.). In the case of the San Francisco Giants, dynamic pricing is being implemented to allow them to dynamically adjust prices by weighing ticket sales data, weather forecasts, upcoming pitching matchups and other variables to help decide whether the team should raise or lower prices right up until the day of the game.

The problem with dynamic pricing is that it doesn’t enable organizations to truly understand the needs, preferences and spending propensities of each and every customer they serve. For example, the problem I see with dynamic pricing for baseball franchises is that it relies on a basic set of variables (e.g. weather, starting lineup, etc.) to determine how to price to the masses, instead of focusing on – and pricing to – each customer’s specific needs. Let’s say I want to go to a baseball game on my birthday. Will the dynamic pricing system offer me a discounted ticket (or should it predict that I am more spendthrift on my birthday)? If my favorite pitcher is starting will the system recognize my willingness to pay more and increase my ticket price? If I regularly attend games throughout the season will the system consider my loyalty and offer me discounts to other games? The respective answers are no, no and no. The advantage here clearly goes to scalpers, as they can still adjust and negotiate prices with each customer they interact with directly. However, where I see the limitations of dynamic pricing end, the benefit of revenue optimization begins.

Revenue optimization technology can provide baseball franchises an accurate picture of demand down to the single customer-level, where the software can codify each customer’s preferences and adjust prices according to their needs, total amount spent and even longevity as a fan (i.e. brand loyalty). Baseball teams already capture tons of customer data through the MLB web portal, where fans can upload and track their favorite teams/players and purchase tickets and merchandise. All of this data can be mined to figure out each customer’s specific price point for every seat of every game! The technology enables baseball franchises to increase ticket sales volume for less popular games, reduce the number of tickets resold in the secondary market, and increase profits for every game. In addition, baseball teams can begin to cross-sell other items like concessions and merchandise to these loyal fans, or even optimize the sale of bundled tickets and/or merchandise. With this increased ability to effectively market to each fan, baseball franchises will become more adept at selling tickets than the scalpers and can soon “steal” their profits back – forcing scalpers to buy tickets if they want to see Major League Baseball’s most prolific stealers.

At the same time, though, a second major event appears to be unfolding:  the emergence of the economics of “free,” or “freeconomics” as provocatively described by Chris Anderson of Wired magazine in his recently published book “Free: The Future of a Radical Price.”  “Free” in Anderson’s formulation is the notion that the near-zero cost of doing business online turns upside down the conventional notion of economics as the science of parsimonious choices under conditions of scarcity.  The “economics of abundance” in Anderson’s phraseology may filter through the prism of our traditional understanding of markets as being good news for cash-strapped consumers (more stuff for which I don’t have to pay money) and bad news for suppliers of goods and services (“free” doesn’t sound like a price that will shore up my profit margins).

But is that right?  I would argue differently: the playing field where Freetopia meets the Great Deleveraging presents unique opportunities for enterprises that are able to use scientific methods to figure out the detailed contours of this new environment.  Household dollars are hard to come by.  But there are other things of value that factor into Freetopian economics: things like time, attention and reputation.  The key challenge for organizations is to figure out what these things are, who cares about them, where they fit into the picture and how to quantify them for optimal outcome.

I distill the following principal arguments from Anderson’s work: (a) the cost of doing business online is nearly zero; (b) transactions in Freetopia are not classical binary exchanges between a single buyer and a single seller, but rather involve a mix of parties where the exchange of cash is only a part of the value equation; and (c) some of the parties to the transaction are willing to offer some things for free in exchange for other things that confer some other value notion.  These complex multiparty transactions involve exchanges of product, service, cash, convenience, labor, information, gifts, reputation and awareness. In other words, Freetopia is not synonymous with free lunch (though, enjoyably, we discover in Anderson’s book the origins of this phrase as a value proposition used by San Francisco saloons in the late 1800s: anyone paying for a beer got a “free” lunch to go with it).

What this prompts us to do is to think in new ways about how our customers’ demand curves fit into that complex web of interests.  What are the components of the total cost borne by the customer in any given transaction, and what are the terms of value?  How valuable to the customer is a reduction in the cost of search?  What would induce the customer to pay more for A while getting B and C for nothing, or perhaps bartering a service (such as writing a review or filling out a questionnaire) that would benefit some other party to the transaction who would then subsidize part of the cash price of A to make it more appealing to the customer?  These are the types of opportunities that emerge on this new playing field.

The added complexity posed by these non-traditional transaction webs suggests that going by gut instinct alone will not suffice for organizations trying to figure out how to optimally supply their customers’ demand curves.  Nor, however, will the methods embedded in earlier generations of revenue optimization solutions be up to the task.  As Freetopia moves more into the mainstream of our economic lives the scientific methods that help us uncover the most important insights will need to do more than apply conventional optimization algorithms to historical daily prices.  At Sentrana our focus is on achieving mastery at the micromarket level – disentangling all the variables that connote what matters to a given customer at a given node in a given transaction opportunity.  As we look into the kind of future that Freetopia presages, we see an increased urgency for nuanced clarity and a growing role for scientific micromarketing – not as a one-off management tool but something at the strategic core of making the most from the opportunities this daunting – but potentially lucrative new world – will provide.

The problem many corporations frequently suffer from is fractured pricing policies where disparate departments within the organization have conflicting rules regarding pricing strategy. This is often a result of unimpeded change within each department, where every manager relies on their own gut instincts at pricing based on their limited view of the ever-changing picture of customer demand. In addition to this proliferation of pricing policies with the potential to impact the market’s demand, other departments in the organization are also making demand-impact decisions, such as advertising and product mix. These practices are often left to chance because most leaders A) don’t realize the problem exists, B) are currently surviving this economy with a meager profit that is most often derived from a “survivalist” measure like cost cutting, layoffs, and running tighter operations, etc., C) are consumed by the sheer volume and complexity surrounding marketing decisions due to the proliferation of advertising channels, products, customers, and supplier networks, or D) if they realize there is a problem, they aren’t aware of what solutions may exist. What these business leaders don’t realize is that they’re leaving enormous profits on the table all the while giving competitors the opportunity to lure their customers away with the “right” price.

To help shed light on the problems these business leaders are facing, I reflect on a quote from John Wooden – one of the most respected college basketball coaches of all-time with 10 NCAA basketball championships during his tenure at UCLA – where he said, “Before you can lead others, you must be able to lead yourself.” This brilliant insight by a legendary sports icon can also serve as an invaluable business axiom: you can’t lead your market until you lead your organization. Simply put, companies – especially those struggling in this economy – should turn their attention inward. Doing so will require new thinking, advanced technology and a change of focus towards effectively generating growth organically (as opposed to via manic serial mergers and acquisitions) for your firm. Forget about the credit crunch (i.e. insufficient investment capital, the dried up commercial-paper market, etc.), falling consumer demand and other external factors that you can’t control.

I believe that in order for companies to be profitable in this economy, they need to adopt both an enabling technology and decision-making infrastructure to help them determine the optimal prices, marketing mix, and product assortments that will maximize revenue and profitability. Successful implementation requires leadership from the executive suite and bottom-up support from the people who work in all functions throughout the organization. Technology is a necessary component and must be able to serve the firm across its organizational silos in ways that allow each part of the organization to achieve optimal productivity and profitability, taking into account both departmental and firmwide objectives. Visibility throughout the organization is also essential, where the stewards of each of the demand and supply levers in effect have the ability to discover previously unknown or misunderstood elasticities – this can lead to a virtuous loop of discovery and profit.

Times like these require business leaders to move past temporary measures to be successful in both the short and long-term – to not only help their companies survive this downturn, but also to be visionary and lead their market! The insight a company requires to effectively manage their marketing decisions should be robust and holistic – not only do they need optimal prices, but also recommendations on products for customer penetration (cross-selling), optimal deals and promotions and guidance to curtail customer churn. These organizations that are able to not only manage, but also execute their ever-growing array of marketing decisions will have the ability to drive their markets, rather than being driven by them, and will be better positioned to continue leading their markets when the economy returns with vigor, as they will be more adept to discover, view, analyze and act upon the opportunities present in their demand environment. I can only assume that the shrinking credit supply, weak consumer demand and other external factors will likely be with us for some time, but they are of little relevance to any business leader looking to grow their firm organically, and seek to embody John Wooden’s principles in the process.

An astute reader pointed out that solving this problem depends at least as much on software as hardware. Indeed, it is a deft blend of hardware as well as software optimization that provides the best hope for making vast reams of information imminently accessible. At the core of this information accessibility problem lies the most inescapable of all culprits – the Laws of Physics. Although the laws are immutable, there is hope.

No matter how fast the chips powering our computers become, there is a bottleneck between hard disk storage and main memory, or RAM. This condition is referred to as being “I/O bound” (I/O stands for input/output – essentially how fast information can be transferred from the disk to the processing units in a computer). Within a computer’s main memory, all activities are performed electronically, which essentially means varying levels of “rather fast.” The major disadvantage of typical disk storage systems is that reading information from them requires mechanical motion (full disclosure: this paradigm of mass storage is currently facing a major disruption from Solid State Disk (SSD) drives, although SSD drives are significantly more expensive today).

The Culprit - Read Head on a Hard Disk Drive

This mechanical motion significantly increases the time it takes to read and write information, slowing system performance. The computer’s throughput is thus bound by not by how fast electrons move, but by how fast the disk can rotate and the disk-head repositioned – to the tune of roughly 200 MB per second. We are physically bound to this mechanical limit, but through some clever tricks ranging from the conventional to the profoundly innovative can give us a quantum leap in the rapid accessibility of information.

Step 1: Cheat the Laws of Physics

If we are ever going to use data to make analytically-driven business decisions, we have to get into the technical weeds just a bit. A good first step is for us to put all of the information in our enterprise databases into what is referred to as a “RAID stack.” RAID stands for Redundant Array of Independent Disks. What this allows us to do (specifically, in what is termed a “RAID-0” configuration), is break up blocks of data and spread them across multiple disk drives. Breaking up information in this manner greatly improves I/O performance by distributing the load across many channels and drives. For reasons we won’t go into here, there is a limit to how many disks we can distribute data across and still get the desired results. Ultimately the attainable performance improvement tops out at 1.6 GB/s (8 disks at 200 MB/s).

Striping Data in a RAID-0 Array

The computer’s RAM has a read/write speed of around 7 GB/s, but if we can only write information into memory at 1.6 GB/s, then we are still under-saturating RAM and the CPU. The CPU can perform calculations at roughly 10 GB/s, which means that a dual quad-core chip architecture (8 cores) allows for 80 GB/s of calculations. We are still not close to maximizing the computer’s potential throughput. Whether or not we can devise a way to maximize the potential of our machines, however, is the difference between only having the bandwidth to perform deep analytics on the top 10% of your product catalog on the one hand, and being able to quickly analyze not only all of your products, but all possible customer-product combinations. Optimizing your business, in fact, depends on the ability to overcome this challenge.

A second way to circumvent I/O boundedness is through compression. By compressing the text stored in the database tables, we can easily achieve a 5:1 gain in throughput as well. This means that the 1.6 GB that has to be picked up by the disk head now gets unpacked into 8 GB by the CPU (although the performance cost of decompression can be high in some cases, there are advances that we can use to get us past this hurdle as well). So by combining two well-known tools right off the bat, we have already achieved a performance improvement of several orders of magnitude compared to how much time it used to take to get 8GB of data to the CPU. Think about how much more of your customer data this allows you to analyze in the same amount of time. But we still have a long way to go before we max out how much information the computer is capable of processing.

Step 2: Cheat the Limitations of Storage

Let’s say that over a two-year span, a company records 25 million individual transactions in its database, and that we are interested in knowing the total sales it made of a specific item: SKU5893. This section describes how a typical database would go about answering that query.

The manner in which we typically store data is not always the most conducive for high-performance computing. Most databases store information as collections of rows. Each row denotes a single unit of interest, such as a sales transaction. Each row has a number of columns that describe the attributes of that transaction, such as the data, customer’s name, item purchased, price, and so on. This can present some problems when data needs to be accessed in certain ways. If you wanted to find all the records involving a specific SKU, it might require scanning across 20 or more columns before getting to the column you need. The value in that column then has to be checked against the desired value to see if it matches, and then this search-and-check process is repeated for the next row until we have searched through the entire table. In massive data sets, all of this search time adds up and creates a crippling performance bottleneck. But there is a way out.

If it seems to you that there has got to be a way to store sales data more manageably, right? By taking advantage of what is called vertical fragmentation, we can do just that. Imagine that our transaction table only has five fields: Location, Customer Name, Item, Price, and Date, as in the example below.

Row-Based Table

With tens of thousands of transactions each day, this table quickly accumulates a lot of rows. However, if we decide to orient this table by its columns instead of its rows, we would get the following:

We now have five two-column tables after adding a unique id field to each column that maps back to the information for each row. Several important results come from this new orientation. First, notice how much repetition of data we have in certain tables such as Date and Location. For instance, the value “1/1/2007” will be repeated thousands of times in this table. The gains we can achieve by compressing vertically fragmented tables far exceed what we can achieve with row-based tables because run-length encoding (and other compression techniques) because the data model better supports it. The second crucial point is that vertical fragmentation enables us to send to the CPU only the information that it needs to see. The disk-head does not need to scan across columns of data that it doesn’t need – so the 200MB/s that it is able to read is focused only on the column necessary for the query. Tack on another several orders-of-magnitude in I/O improvements.

Step 3: Cheat the Laws of…Math

Now we’re getting somewhere. Data striping, compression, and vertical fragmentation provide a huge boost to the volume of information that we can access and process – indeed we are now getting to and probably exceeding the CPU’s number-crunching ability of 80GB/s. This brings us to our final bottleneck: we can’t speed up how quickly the chip can do the math required for heavy BI analytics.

The solution lies not in the CPU, but in many CPUs. Distributed computing allows us to bring more CPUs into the mix while also feeding them from their own disks – what is referred to as a “shared nothing” architecture. If you use ten machines, you can distribute a a billion rows of sales data evenly across all ten machines, leaving 100 million rows on each. Now each machine is executing at warp speed on only a portion of the database, completing our search for sales of a specific SKU in a fraction of the normal processing time.

Integrating all of the techniques that we have covered can be summed in the following steps:

1. Distribute a database across multiple machines, so that each gets some fraction of the total number of rows in the original
2. On each machine, vertically fragment the database section stored on it
3. Stripe the data across multiple independent disks on each machine
4. Finally compress that data,

We have now achieved massively parallel and high performance processing. Each machine now runs through all of the information it has and sends the relevant information to the CPU, which can finally hit the processing limits of current chip architectures. The I/O curse long since in our rear-view mirror, we can finally begin to unlock the incredible amount of information latent in our very own data.

Sentrana’s Role

I should note that the above story is an idealization. Real-world implementation of the interconnecting innovations that I have outlined here confront serious challenges of their own: decompression taxes the CPU; RAID is expensive and the energy drain of all those disks can be significant; management of multiple machines that are co-operating in a cluster is itself a sophisticated systems administration task, and the list goes on. Though valid concerns all, the more important point is that these are all solvable problems. These real-world implementation problems are the ones that Sentrana’s research has focused on in order to put business intelligence and predictive analytics at a person’s fingertips. Truly great business insights are like scientific discoveries – they stem from first asking an important question and then breaking it down into manageable pieces so that it can be answered. In order to support those moments of intuition in which the momentous questions are first asked, we have to be as fast as that other great computer at every decision-maker’s disposal.

Let’s say hypothetically that this printer has 10 different attributes like the ones mentioned above on which every buyer places a value, even it happens to be zero. There is a segment of the printer-buying population that wants all 10 attributes, including the HP brand name of the seller, and that segment is willing to pay a higher price. No other seller can satisfy all 10 attributes, giving HP a monopoly on that attribute set. But as a seller, HP operates within constraints since other sellers offer the same exact printer at a lower price in return for providing fewer attributes. Thus, HP cannot set its prices as a pure monopolist, because an excessively high price will drive too much of the market to the next lowest price tier. HP’s competitive position is what I call a micro-monopoly (or “Micropoly” if you prefer the conflation, as I do). The explanation for this price dispersion is that every seller of this printer satisfies a unique mix of attributes demanded by a particular segment of the market. For that segment, the seller has a limited amount of micro-monopoly pricing power.

When viewed from this angle, it becomes easy to see why it makes more sense to set prices based on what your customers value rather than what your competitors charge. The reason is that one firm may compete only tangentially with another firm that sells the same products. The obvious question then is what happens when two firms fulfill the same exact mix of attributes. At this point, firms would then compete on price, but I think this logical extension can be somewhat misleading. In the real world, no two firms ever truly occupy the same attribute space. There will always be at least some differences in the total experience and feel that the customer gets from making the purchase, and thus the potential for price differentiation exists. Multiple optimum prices for the same product can exist in the marketplace. A profit maximizing firm’s objective should not be to race to the bottom in a low price battle with competitors, but rather to understand very clearly what its price ceiling is.