Building a robust data management environment is in many ways like building a house. There are three components to building a good house. First of all, there are some fundamental questions you need to ask before doing anything. Why are you building the house in the first place? What are the important goals and benefits you want to enjoy? What other things are you willing to trade off to realize those benefits? Asking and answering those questions will help with the second component: building a model, or architectural blueprint. There are many different ways to build a house (or a data management system). Not all of them will be right for the needs you have in mind. There are efficiencies to designing and building in certain ways – and, as always, there are trade-offs with any given choice. Finally, once you have established a workable model, it’s time to build out the infrastructure. That starts with the plumbing. Nothing else in the house is going to work well without good plumbing which, seamlessly and unobserved, harnesses the flow of water (or data, in our analogy) to efficient uses as and when needed. Then comes the foundation – the platform to support the house according to your model. Think of the plumbing and the foundation as the transmission pipes, the controls to regulate the flow of information, the storage repositories and the other critical supports for your data management platform.

robust systems need good blueprints

Asking the Questions that Matter for You

It’s hard to imagine that someone would build a home without first asking and answering some basic questions about what purpose the home is meant to serve. But all too often enterprise managers think of their data intelligence needs in terms of generic, one-size-fits-all products and solutions. They may be driven by the perceived urgency of getting immediate results, so they do not put the extra time into thinking through all the details that have to be in place in order for a solution to best meet their targeted needs. They build up organizational IT resources but fail to adequately integrate these resources into business decision-making processes so that business goals and technological capabilities are aligned. By not asking the right questions up front, managers increase the likelihood that their IT investment will fail to achieve the specified goals.

The place to start is by asking what kind of business you are. What are the core elements of your business strategy? What kind of information – about customers, products, channels, territories, campaigns, transaction dates and so on – do you need to pull into your organization every day in order to make decisions that help accomplish that strategy? What performance measures do you seek to optimize – for example gross margin, market share or something else? Is there a single overriding imperative for all territories or are there more nuanced considerations for local markets? Who needs access to the data, both within the organization and among trade partners? These broad-based questions can then lead into more detailed questions around specific tactical opportunities that can help support the overall strategic goals.

Modeling Around the Right Questions

One homebuilder wants a large kitchen with lots of counter space for preparing gourmet meals. Another wants to optimize for green initiatives –creating spaces for solar panels and the like. Every homebuilder needs an architectural blueprint and model that defines the best tools, materials and processes to accomplish the job at hand. These decisions should flow naturally from the questions asked and answered in the previous stage.

Data models are designed for different purposes. Some place emphasis on providing a continual stream of information for decision-making in real time. Others focus on high quality analytical tools applied to batch data that are aggregated for a particular time period – for example daily, weekly or monthly. A critical driver of data architecture is the volume of information the system is expected to handle and the types of activities users expect to be performing when they access the databases. Information processing has a cost, which includes a time component and a money component. Thinking back to the homebuilder who wants a state-of-the-art kitchen space, the architectural  blueprint for that kitchen has to think through all the activities our homebuilder-chef is going to perform – how to optimize the process of retrieving ingredients, setting up preparation work stations, pulling out pots and pans and chef’s knives, and cooking on the stove or in the oven. The data architecture similarly has to visualize the user retrieving various bits of data from different locations at different times for different purposes, and figure out the configuration that works best.

Laying Pipes and Building a Platform

The most elegant model in the world will be of little use if the infrastructure is not solid. Living in a house we tend not to think about the plumbing as long as it is working – we wash dishes, water the plants and fill the ice trays without wondering about how the system works. But we will certainly notice when it doesn’t work – when water doesn’t  come out of the faucet or when it floods the basement. Likewise a business user at a computer workstation is probably not going to think about how all that information flowed from somewhere out there to show up at her desk – unless it doesn’t show up or is in an unusable format. Laying the transmission conduits and controls that regulate the flow of data to where, when and how it is needed in the organization is a serious undertaking.

When information comes into the organization from “out there” it is unfortunately a bit more complex in form than the municipal water that enters your home’s plumbing system. It is easily corruptible and subject to misuse if robust protocols and procedures are not in place to keep it clean and usable. In a surprisingly large number of cases data “formats” consist of little more than unprotected Excel spreadsheets with inconsistent definitions and naming conventions for product categories, subsets, SKUs and customer types. That will likely prove to be of little use in facilitating actionable analysis. To be usable data need to be clean and granular – in other words, accurate, in the right format and at the right level of detail for the analysis to be performed.

Organization Matters, Too

Business managers do not always take into account the organizational aspect of data management. The solutions architecture will not be effective if the enterprise is unable to successfully weave it into the fabric of the organizational culture and existing decision making processes. Empirical evidence suggests that rigorous analytical methods play a relatively minor role in the decision making practices of many organizations – especially those at higher strategic levels but also those related to daily tactical opportunities. Ask yourself how different levels of decisions are made in your organization. What role, if any, does data analysis play? How persuasive are data-supported arguments – for example in comparison with rigid top-down planning imperatives, an ad hoc “firefighting” culture or political and personality-centered negotiations? What needs to change in terms of both formal processes and the informal cultural context for actionable solutions architecture to make a lasting impact and lead to more informed decisions?

Like a house, a data management system is best if it can last for a long time and have the flexibility to adapt to changing needs and circumstances. Business strategies will change, as will technology. Far better that a system can adapt to this change without requiring an entire rebuilding effort from scratch. Start with the right questions, and you have a much better chance of building a data management capability that can successfully evolve with your organization.

For manufacturers of foodservice products an important and often elusive goal is to gain visibility into the factors shaping and influencing downstream demand. The view from upstream is obscured by one or more layers of intermediation separating products from their end customers. Manufacturers typically set aside the largest part of their sales and marketing budgets for payments to trade partners, but evidence suggests that these expenditures do little to improve their understanding of actual downstream demand. Whether on their own or in collaboration with trade partners, manufacturers need to make better use of the data that can provide accurate intelligence about what is happening downstream. The good news is that the data are available, and new technologies are converging to enable manufacturers to capture information from which to make better sales & marketing decisions. The challenge is to get around the obstacles that are preventing this from happening.

social networking is changing the way people decide where, when, what and with whom to eat

The end point of the foodservice value chain, of course, is the patron. In a perfect world restaurant operators would be able to predict how many patrons are showing up at their doors every night of the week, and what quantities of which menu offerings they would be ordering to eat and drink. They could then place orders upstream in total accordance with this knowledge. The world is not perfect, though, and no restaurant owner has such powers of clairvoyance. What they do have, increasingly, is the benefit of something that barely existed just a few years ago – social networking technology. More and more of the decisions that people make on a daily basis – including where, when and with whom to eat meals – are utilizing the many technological tributaries of the social networking phenomenon. For their part, restaurant operators have also been figuring out how to get into this game. Even the independent local establishments that lack the large corporate infrastructure of national chains are able to use these new digital tools to better understand their own markets and to control their own marketing – to reach out to potential diners with offers and other targeted inducements to patronize their tables. In so doing they are gaining actionable insights with which to better estimate demand. In turn, they can place orders to their suppliers with a higher level of confidence that these orders are a reasonably accurate reflection of what their customers will actually be ordering, when and in what quantity.

The presence of social networking technology is increasing the quantity and quality of digital signals coming from downstream markets. But there is another rapidly evolving technology that has the ability to amplify these signals even more powerfully up the value chain. That is predictive technology for sales and marketing decisions. Predictive technology applies rigorous quantitative methods to arm sales and marketing decision makers with recommendations to target the right customers with the right products, promotions, pricing and timing. This approach feeds on data signals – the more the better across as many customer-product combinations as possible – and helps to untangle the many diverse factors affecting demand at a very granular level. The algorithms that power predictive technology solutions can capture the digital demand signals coming from operators and patrons downstream, analyze them and provide useful intelligence to managers in corporate headquarters as well as sales professionals on the front line – information that can help them make decisions based on more than just guesswork. This can result in a higher likelihood of success for every offer presented to a customer.

This convergence of two powerful technology trends offers potential benefits throughout the foodservice value chain. For manufacturers the big question – going back to the discussion in the opening paragraph of this post – is how to get past the obstacles that impede the view downstream. There are different ways of doing this, and best practices are likely to evolve over time. One path to greater demand clarity can occur through better and more comprehensive collaborative processes with trade partners. Midstream distributors have their own set of challenges in the current environment – for example managing profit margin stability under conditions of unusually high volatility in week-to-week revenue movements alongside steadily increasing COGS trends for key products and categories. More accurate forecasting for price and other demand levers – assortment, promotions and sales force effort – can result in more predictable margins without the wild fluctuations seen today. That in turn can provide an incentive for distributors to be willing to look at alternative models to the stultifying “pay to play” template for trade spend that prevails today – including closer upstream collaboration.

Thanks to the growing capabilities afforded by social networking technology restaurant operators are steadily improving their ability to understand the intricacies of their markets and plan their foodservice purchases accordingly. Their upstream suppliers in turn can apply predictive technology solutions to identify where their promotions, sales efforts and deal bundling will most likely fall on the most receptive ears. Manufacturers who figure out how best to tap into these more accurate and better-amplified data signals will likely be rewarded with a clearer and farther view downstream.

To answer this question, let’s begin briefly by dissecting what IBM has introduced. Imagine that you are receiving a continuous stream of data, such as stock prices on the Nasdaq. These figures must be quickly analyzed so that the proper buy and sell orders can be placed. Suppose that you also need to base your decisions not just on the Nasdaq prices but also the numbers figures coming in from dozens of other exchanges.

No Match for the Information at Hand

Just for fun, let’s say that you also want to include up-to-the-second weather information of 20 cities in your decision process. We can safely say that there are not a great deal of human beings in the world that could look at all of this information in the blink of an eye, make the best possible decision, and then repeat the exact same process a split-second later with new information. Well, this example highlights the best application for stream computing. In the stock price analysis outlined above, we needed to evaluate information coming to us on a continuous basis from a number of different sources, evaluate it, and deploy the result in a desired manner. Let’s say that we wanted analyze the correlation between Motorola’s stock price movements and the price movements of 50 other stocks on 50 different exchanges, as well as the outdoor temperatures of every single world capitol. IBM’s breakthrough allows you to do this and also update your analysis in real-time as new information is received. That’s the good news.

The bad news is that there are real limits to the insight that can be obtained this way. There are really two areas in which stream computing can be applied: link detection and itemset detection. These two are related – link detection tries to identify events that are correlated, and itemset detection tries to identify events that are co-incident (i.e. the prices of GE and Sony stock went up, AND the price of Motorola went up).

Not Quite Forward-Looking

Itemset detection algorithms have progressed to the point where they can be accurate and effective without taxing the computer’s resources too heavily. IBM’s accomplishment lies in deploying this algorithm for real-time itemset detection on unprecedented scale, but ultimately, the business insight that can be extracted from correlation and co-incidence is limited. Frankly, IBM’s press release statement that System S allows users to “…create a forward-looking analysis of data from any source,” is a bit of a fairy tale. Any competent manager will tell you that “forward looking analysis” based only on correlations in historical data is not a recipe for success. With itemset detection (which is closely related to and sometimes confused with association mining), you can now look at more data more quickly than ever before, but you can’t ultimately do much new with it.

Truly forward-looking analysis requires going several steps beyond itemset detection, because making accurate predictions about events that have never happened requires that you do more than analyze past correlations. For example, how could you predict the highest price at which a customer would be willing to buy a product that they have never purchased before using only their previous purchase information? Or how about predicting the effect an advertisement will have in a market that you have never advertised in before? Integrating techniques that are better geared for this analysis such as principal components analysis, collaborative filtering, and others make these types of predictive analysis possible, not to mention significantly more robust than correlative analytics. The downside of integrating of all of these techniques into your analysis is that it becomes extremely expensive computationally. Other related breakthroughs are required to make this feasible in a business setting (see Cheating Your Way Into Business Visibility). To make better informed decisions with an understanding of the likelihood of possible outcomes, managers and policymakers alike need a way to reduce the uncertainty inherent in them. The aforementioned “virgin sale” process is a very micro-example that exemplifies this problem. Sadly, being able to comb through data more quickly by itself does not get us to that higher end-state. IBM can claim a notable achievement in computing, but hailing it as a quantum leap for businesses decision support understates the true challenge of making better business decisions.

An astute reader pointed out that solving this problem depends at least as much on software as hardware. Indeed, it is a deft blend of hardware as well as software optimization that provides the best hope for making vast reams of information imminently accessible. At the core of this information accessibility problem lies the most inescapable of all culprits – the Laws of Physics. Although the laws are immutable, there is hope.

No matter how fast the chips powering our computers become, there is a bottleneck between hard disk storage and main memory, or RAM. This condition is referred to as being “I/O bound” (I/O stands for input/output – essentially how fast information can be transferred from the disk to the processing units in a computer). Within a computer’s main memory, all activities are performed electronically, which essentially means varying levels of “rather fast.” The major disadvantage of typical disk storage systems is that reading information from them requires mechanical motion (full disclosure: this paradigm of mass storage is currently facing a major disruption from Solid State Disk (SSD) drives, although SSD drives are significantly more expensive today).

The Culprit - Read Head on a Hard Disk Drive

This mechanical motion significantly increases the time it takes to read and write information, slowing system performance. The computer’s throughput is thus bound by not by how fast electrons move, but by how fast the disk can rotate and the disk-head repositioned – to the tune of roughly 200 MB per second. We are physically bound to this mechanical limit, but through some clever tricks ranging from the conventional to the profoundly innovative can give us a quantum leap in the rapid accessibility of information.

Step 1: Cheat the Laws of Physics

If we are ever going to use data to make analytically-driven business decisions, we have to get into the technical weeds just a bit. A good first step is for us to put all of the information in our enterprise databases into what is referred to as a “RAID stack.” RAID stands for Redundant Array of Independent Disks. What this allows us to do (specifically, in what is termed a “RAID-0” configuration), is break up blocks of data and spread them across multiple disk drives. Breaking up information in this manner greatly improves I/O performance by distributing the load across many channels and drives. For reasons we won’t go into here, there is a limit to how many disks we can distribute data across and still get the desired results. Ultimately the attainable performance improvement tops out at 1.6 GB/s (8 disks at 200 MB/s).

Striping Data in a RAID-0 Array

The computer’s RAM has a read/write speed of around 7 GB/s, but if we can only write information into memory at 1.6 GB/s, then we are still under-saturating RAM and the CPU. The CPU can perform calculations at roughly 10 GB/s, which means that a dual quad-core chip architecture (8 cores) allows for 80 GB/s of calculations. We are still not close to maximizing the computer’s potential throughput. Whether or not we can devise a way to maximize the potential of our machines, however, is the difference between only having the bandwidth to perform deep analytics on the top 10% of your product catalog on the one hand, and being able to quickly analyze not only all of your products, but all possible customer-product combinations. Optimizing your business, in fact, depends on the ability to overcome this challenge.

A second way to circumvent I/O boundedness is through compression. By compressing the text stored in the database tables, we can easily achieve a 5:1 gain in throughput as well. This means that the 1.6 GB that has to be picked up by the disk head now gets unpacked into 8 GB by the CPU (although the performance cost of decompression can be high in some cases, there are advances that we can use to get us past this hurdle as well). So by combining two well-known tools right off the bat, we have already achieved a performance improvement of several orders of magnitude compared to how much time it used to take to get 8GB of data to the CPU. Think about how much more of your customer data this allows you to analyze in the same amount of time. But we still have a long way to go before we max out how much information the computer is capable of processing.

Step 2: Cheat the Limitations of Storage

Let’s say that over a two-year span, a company records 25 million individual transactions in its database, and that we are interested in knowing the total sales it made of a specific item: SKU5893. This section describes how a typical database would go about answering that query.

The manner in which we typically store data is not always the most conducive for high-performance computing. Most databases store information as collections of rows. Each row denotes a single unit of interest, such as a sales transaction. Each row has a number of columns that describe the attributes of that transaction, such as the data, customer’s name, item purchased, price, and so on. This can present some problems when data needs to be accessed in certain ways. If you wanted to find all the records involving a specific SKU, it might require scanning across 20 or more columns before getting to the column you need. The value in that column then has to be checked against the desired value to see if it matches, and then this search-and-check process is repeated for the next row until we have searched through the entire table. In massive data sets, all of this search time adds up and creates a crippling performance bottleneck. But there is a way out.

If it seems to you that there has got to be a way to store sales data more manageably, right? By taking advantage of what is called vertical fragmentation, we can do just that. Imagine that our transaction table only has five fields: Location, Customer Name, Item, Price, and Date, as in the example below.

Row-Based Table

With tens of thousands of transactions each day, this table quickly accumulates a lot of rows. However, if we decide to orient this table by its columns instead of its rows, we would get the following:

We now have five two-column tables after adding a unique id field to each column that maps back to the information for each row. Several important results come from this new orientation. First, notice how much repetition of data we have in certain tables such as Date and Location. For instance, the value “1/1/2007” will be repeated thousands of times in this table. The gains we can achieve by compressing vertically fragmented tables far exceed what we can achieve with row-based tables because run-length encoding (and other compression techniques) because the data model better supports it. The second crucial point is that vertical fragmentation enables us to send to the CPU only the information that it needs to see. The disk-head does not need to scan across columns of data that it doesn’t need – so the 200MB/s that it is able to read is focused only on the column necessary for the query. Tack on another several orders-of-magnitude in I/O improvements.

Step 3: Cheat the Laws of…Math

Now we’re getting somewhere. Data striping, compression, and vertical fragmentation provide a huge boost to the volume of information that we can access and process – indeed we are now getting to and probably exceeding the CPU’s number-crunching ability of 80GB/s. This brings us to our final bottleneck: we can’t speed up how quickly the chip can do the math required for heavy BI analytics.

The solution lies not in the CPU, but in many CPUs. Distributed computing allows us to bring more CPUs into the mix while also feeding them from their own disks – what is referred to as a “shared nothing” architecture. If you use ten machines, you can distribute a a billion rows of sales data evenly across all ten machines, leaving 100 million rows on each. Now each machine is executing at warp speed on only a portion of the database, completing our search for sales of a specific SKU in a fraction of the normal processing time.

Integrating all of the techniques that we have covered can be summed in the following steps:

1. Distribute a database across multiple machines, so that each gets some fraction of the total number of rows in the original
2. On each machine, vertically fragment the database section stored on it
3. Stripe the data across multiple independent disks on each machine
4. Finally compress that data,

We have now achieved massively parallel and high performance processing. Each machine now runs through all of the information it has and sends the relevant information to the CPU, which can finally hit the processing limits of current chip architectures. The I/O curse long since in our rear-view mirror, we can finally begin to unlock the incredible amount of information latent in our very own data.

Sentrana’s Role

I should note that the above story is an idealization. Real-world implementation of the interconnecting innovations that I have outlined here confront serious challenges of their own: decompression taxes the CPU; RAID is expensive and the energy drain of all those disks can be significant; management of multiple machines that are co-operating in a cluster is itself a sophisticated systems administration task, and the list goes on. Though valid concerns all, the more important point is that these are all solvable problems. These real-world implementation problems are the ones that Sentrana’s research has focused on in order to put business intelligence and predictive analytics at a person’s fingertips. Truly great business insights are like scientific discoveries – they stem from first asking an important question and then breaking it down into manageable pieces so that it can be answered. In order to support those moments of intuition in which the momentous questions are first asked, we have to be as fast as that other great computer at every decision-maker’s disposal.

Not so, in fact. When you start asking extremely detailed questions that require combing through years of detailed historical data and then performing mathematical transformations on each of those figures, you will find out rather quickly the limits of processing speed when your results finish compiling in a week or so. The thing is that most of us never push up against the processing speed frontier. We can see that every year computers get faster, chips get smaller, and Excel seems to have more rows. Moore’s Law prevails. The trouble is that all the while the rate at which the data universe expands is screaming past advances in processing capabilities, and that rate does not fluctuate with the economic downturn. Consider the markets for microprocessors, which allow us to perform those calculations and manipulate data, and hard drives, which allow for storage of data. Microprocessor sales have been dealt a sharp blow by the global downturn as computer sales have slowed, but worldwide shipments of hard disk drives (HDDs) roughly maintained 2007 levels even in the worst quarters of the recession (and the drives themselves contain more memory). Solid state and flash memory shipments were down, but the evidence suggests that this is due to consumers substituting HDDs for other types of memory, rather than simply not storing more information. The demand for data storage, while not completely recession-proof, is nonetheless of the hardier variety.

Simply put, information of all kinds accumulates faster than we can analyze it. We are losing the race, and the gap is widening, not shrinking. As for what this ultimately means, I will now make a rather dour point. A fashionable explanation for the recession among both politicians and many “Main Street” types is that greed is what did us in. The greed of the bankers, the hedge funds, the fat cats, the small cats, whomever – greed is the culprit. But that doesn’t explain everything by a long shot. Even the greediest person doesn’t want the party to end and the money to stop coming in. Might it be possible that they weren’t able to ask the questions that might have led to certain debt instruments having never been created? Financial services employees have more information available to them than decision makers any other industry, and still here we find ourselves. Think about how many times each day similarly misinformed decisions are made inside corporations all across the world. The information is there, but we are more often than not letting it rot on the docks.

What makes the distinction between information and intelligence critical is that information is getting more accessible by the day. This democratization of BI is evidenced nowhere more so than at Microsoft. In 2008, Microsoft unveiled several projects in the late stages of development that it claims will put BI capabilities at the fingertips of more users than ever before. “Project Madison” will massively increase Microsoft’s information storage capabilities, while the “Kilimanjaro” and “Gemini” projects together will provide easy-to-use reporting and analysis tools designed to drastically reduce the complexity of using traditional BI tools – all at very low cost compared to large-scale ERP implementation. The possibilities abound. But I still ask the question: what are all of these newly empowered users going to do with all of this information once they can access it at the push of a button?

I am excited by the idea of so many more information workers being able to ask the questions that end up driving businesses to continuously reinvent and perfect themselves, but I worry about relevance. Will these capabilities actually increase the amount of intelligence available to decision makers? Any business decision can be thought of as a bet that some desired future state will materialize as a result of a present course of action. Business intelligence tools as we know them more often than not do not help us make more intelligent bets when it comes to the future. The problem is that we think they do. More data often makes the task of identifying the true predictors of business success and isolating their effects more difficult. In order for a company to get the most out of its data, it needs PI as well as BI capabilities at the fingertips of decision makers. For marketing and pricing to become a more fact-driven corporate discipline, we must recognize the need not for more data, but ways of evaluating the probability of outcomes based on only the factors that matter. This is not child’s play. Information alone, however well-groomed, is simply not sufficient to meet this need.