To answer this question, let’s begin briefly by dissecting what IBM has introduced. Imagine that you are receiving a continuous stream of data, such as stock prices on the Nasdaq. These figures must be quickly analyzed so that the proper buy and sell orders can be placed. Suppose that you also need to base your decisions not just on the Nasdaq prices but also the numbers figures coming in from dozens of other exchanges.

No Match for the Information at Hand

Just for fun, let’s say that you also want to include up-to-the-second weather information of 20 cities in your decision process. We can safely say that there are not a great deal of human beings in the world that could look at all of this information in the blink of an eye, make the best possible decision, and then repeat the exact same process a split-second later with new information. Well, this example highlights the best application for stream computing. In the stock price analysis outlined above, we needed to evaluate information coming to us on a continuous basis from a number of different sources, evaluate it, and deploy the result in a desired manner. Let’s say that we wanted analyze the correlation between Motorola’s stock price movements and the price movements of 50 other stocks on 50 different exchanges, as well as the outdoor temperatures of every single world capitol. IBM’s breakthrough allows you to do this and also update your analysis in real-time as new information is received. That’s the good news.

The bad news is that there are real limits to the insight that can be obtained this way. There are really two areas in which stream computing can be applied: link detection and itemset detection. These two are related – link detection tries to identify events that are correlated, and itemset detection tries to identify events that are co-incident (i.e. the prices of GE and Sony stock went up, AND the price of Motorola went up).

Not Quite Forward-Looking

Itemset detection algorithms have progressed to the point where they can be accurate and effective without taxing the computer’s resources too heavily. IBM’s accomplishment lies in deploying this algorithm for real-time itemset detection on unprecedented scale, but ultimately, the business insight that can be extracted from correlation and co-incidence is limited. Frankly, IBM’s press release statement that System S allows users to “…create a forward-looking analysis of data from any source,” is a bit of a fairy tale. Any competent manager will tell you that “forward looking analysis” based only on correlations in historical data is not a recipe for success. With itemset detection (which is closely related to and sometimes confused with association mining), you can now look at more data more quickly than ever before, but you can’t ultimately do much new with it.

Truly forward-looking analysis requires going several steps beyond itemset detection, because making accurate predictions about events that have never happened requires that you do more than analyze past correlations. For example, how could you predict the highest price at which a customer would be willing to buy a product that they have never purchased before using only their previous purchase information? Or how about predicting the effect an advertisement will have in a market that you have never advertised in before? Integrating techniques that are better geared for this analysis such as principal components analysis, collaborative filtering, and others make these types of predictive analysis possible, not to mention significantly more robust than correlative analytics. The downside of integrating of all of these techniques into your analysis is that it becomes extremely expensive computationally. Other related breakthroughs are required to make this feasible in a business setting (see Cheating Your Way Into Business Visibility). To make better informed decisions with an understanding of the likelihood of possible outcomes, managers and policymakers alike need a way to reduce the uncertainty inherent in them. The aforementioned “virgin sale” process is a very micro-example that exemplifies this problem. Sadly, being able to comb through data more quickly by itself does not get us to that higher end-state. IBM can claim a notable achievement in computing, but hailing it as a quantum leap for businesses decision support understates the true challenge of making better business decisions.

An astute reader pointed out that solving this problem depends at least as much on software as hardware. Indeed, it is a deft blend of hardware as well as software optimization that provides the best hope for making vast reams of information imminently accessible. At the core of this information accessibility problem lies the most inescapable of all culprits – the Laws of Physics. Although the laws are immutable, there is hope.

No matter how fast the chips powering our computers become, there is a bottleneck between hard disk storage and main memory, or RAM. This condition is referred to as being “I/O bound” (I/O stands for input/output – essentially how fast information can be transferred from the disk to the processing units in a computer). Within a computer’s main memory, all activities are performed electronically, which essentially means varying levels of “rather fast.” The major disadvantage of typical disk storage systems is that reading information from them requires mechanical motion (full disclosure: this paradigm of mass storage is currently facing a major disruption from Solid State Disk (SSD) drives, although SSD drives are significantly more expensive today).

The Culprit - Read Head on a Hard Disk Drive

This mechanical motion significantly increases the time it takes to read and write information, slowing system performance. The computer’s throughput is thus bound by not by how fast electrons move, but by how fast the disk can rotate and the disk-head repositioned – to the tune of roughly 200 MB per second. We are physically bound to this mechanical limit, but through some clever tricks ranging from the conventional to the profoundly innovative can give us a quantum leap in the rapid accessibility of information.

Step 1: Cheat the Laws of Physics

If we are ever going to use data to make analytically-driven business decisions, we have to get into the technical weeds just a bit. A good first step is for us to put all of the information in our enterprise databases into what is referred to as a “RAID stack.” RAID stands for Redundant Array of Independent Disks. What this allows us to do (specifically, in what is termed a “RAID-0” configuration), is break up blocks of data and spread them across multiple disk drives. Breaking up information in this manner greatly improves I/O performance by distributing the load across many channels and drives. For reasons we won’t go into here, there is a limit to how many disks we can distribute data across and still get the desired results. Ultimately the attainable performance improvement tops out at 1.6 GB/s (8 disks at 200 MB/s).

Striping Data in a RAID-0 Array

The computer’s RAM has a read/write speed of around 7 GB/s, but if we can only write information into memory at 1.6 GB/s, then we are still under-saturating RAM and the CPU. The CPU can perform calculations at roughly 10 GB/s, which means that a dual quad-core chip architecture (8 cores) allows for 80 GB/s of calculations. We are still not close to maximizing the computer’s potential throughput. Whether or not we can devise a way to maximize the potential of our machines, however, is the difference between only having the bandwidth to perform deep analytics on the top 10% of your product catalog on the one hand, and being able to quickly analyze not only all of your products, but all possible customer-product combinations. Optimizing your business, in fact, depends on the ability to overcome this challenge.

A second way to circumvent I/O boundedness is through compression. By compressing the text stored in the database tables, we can easily achieve a 5:1 gain in throughput as well. This means that the 1.6 GB that has to be picked up by the disk head now gets unpacked into 8 GB by the CPU (although the performance cost of decompression can be high in some cases, there are advances that we can use to get us past this hurdle as well). So by combining two well-known tools right off the bat, we have already achieved a performance improvement of several orders of magnitude compared to how much time it used to take to get 8GB of data to the CPU. Think about how much more of your customer data this allows you to analyze in the same amount of time. But we still have a long way to go before we max out how much information the computer is capable of processing.

Step 2: Cheat the Limitations of Storage

Let’s say that over a two-year span, a company records 25 million individual transactions in its database, and that we are interested in knowing the total sales it made of a specific item: SKU5893. This section describes how a typical database would go about answering that query.

The manner in which we typically store data is not always the most conducive for high-performance computing. Most databases store information as collections of rows. Each row denotes a single unit of interest, such as a sales transaction. Each row has a number of columns that describe the attributes of that transaction, such as the data, customer’s name, item purchased, price, and so on. This can present some problems when data needs to be accessed in certain ways. If you wanted to find all the records involving a specific SKU, it might require scanning across 20 or more columns before getting to the column you need. The value in that column then has to be checked against the desired value to see if it matches, and then this search-and-check process is repeated for the next row until we have searched through the entire table. In massive data sets, all of this search time adds up and creates a crippling performance bottleneck. But there is a way out.

If it seems to you that there has got to be a way to store sales data more manageably, right? By taking advantage of what is called vertical fragmentation, we can do just that. Imagine that our transaction table only has five fields: Location, Customer Name, Item, Price, and Date, as in the example below.

Row-Based Table

With tens of thousands of transactions each day, this table quickly accumulates a lot of rows. However, if we decide to orient this table by its columns instead of its rows, we would get the following:

We now have five two-column tables after adding a unique id field to each column that maps back to the information for each row. Several important results come from this new orientation. First, notice how much repetition of data we have in certain tables such as Date and Location. For instance, the value “1/1/2007” will be repeated thousands of times in this table. The gains we can achieve by compressing vertically fragmented tables far exceed what we can achieve with row-based tables because run-length encoding (and other compression techniques) because the data model better supports it. The second crucial point is that vertical fragmentation enables us to send to the CPU only the information that it needs to see. The disk-head does not need to scan across columns of data that it doesn’t need – so the 200MB/s that it is able to read is focused only on the column necessary for the query. Tack on another several orders-of-magnitude in I/O improvements.

Step 3: Cheat the Laws of…Math

Now we’re getting somewhere. Data striping, compression, and vertical fragmentation provide a huge boost to the volume of information that we can access and process – indeed we are now getting to and probably exceeding the CPU’s number-crunching ability of 80GB/s. This brings us to our final bottleneck: we can’t speed up how quickly the chip can do the math required for heavy BI analytics.

The solution lies not in the CPU, but in many CPUs. Distributed computing allows us to bring more CPUs into the mix while also feeding them from their own disks – what is referred to as a “shared nothing” architecture. If you use ten machines, you can distribute a a billion rows of sales data evenly across all ten machines, leaving 100 million rows on each. Now each machine is executing at warp speed on only a portion of the database, completing our search for sales of a specific SKU in a fraction of the normal processing time.

Integrating all of the techniques that we have covered can be summed in the following steps:

1. Distribute a database across multiple machines, so that each gets some fraction of the total number of rows in the original
2. On each machine, vertically fragment the database section stored on it
3. Stripe the data across multiple independent disks on each machine
4. Finally compress that data,

We have now achieved massively parallel and high performance processing. Each machine now runs through all of the information it has and sends the relevant information to the CPU, which can finally hit the processing limits of current chip architectures. The I/O curse long since in our rear-view mirror, we can finally begin to unlock the incredible amount of information latent in our very own data.

Sentrana’s Role

I should note that the above story is an idealization. Real-world implementation of the interconnecting innovations that I have outlined here confront serious challenges of their own: decompression taxes the CPU; RAID is expensive and the energy drain of all those disks can be significant; management of multiple machines that are co-operating in a cluster is itself a sophisticated systems administration task, and the list goes on. Though valid concerns all, the more important point is that these are all solvable problems. These real-world implementation problems are the ones that Sentrana’s research has focused on in order to put business intelligence and predictive analytics at a person’s fingertips. Truly great business insights are like scientific discoveries – they stem from first asking an important question and then breaking it down into manageable pieces so that it can be answered. In order to support those moments of intuition in which the momentous questions are first asked, we have to be as fast as that other great computer at every decision-maker’s disposal.

Not so, in fact. When you start asking extremely detailed questions that require combing through years of detailed historical data and then performing mathematical transformations on each of those figures, you will find out rather quickly the limits of processing speed when your results finish compiling in a week or so. The thing is that most of us never push up against the processing speed frontier. We can see that every year computers get faster, chips get smaller, and Excel seems to have more rows. Moore’s Law prevails. The trouble is that all the while the rate at which the data universe expands is screaming past advances in processing capabilities, and that rate does not fluctuate with the economic downturn. Consider the markets for microprocessors, which allow us to perform those calculations and manipulate data, and hard drives, which allow for storage of data. Microprocessor sales have been dealt a sharp blow by the global downturn as computer sales have slowed, but worldwide shipments of hard disk drives (HDDs) roughly maintained 2007 levels even in the worst quarters of the recession (and the drives themselves contain more memory). Solid state and flash memory shipments were down, but the evidence suggests that this is due to consumers substituting HDDs for other types of memory, rather than simply not storing more information. The demand for data storage, while not completely recession-proof, is nonetheless of the hardier variety.

Simply put, information of all kinds accumulates faster than we can analyze it. We are losing the race, and the gap is widening, not shrinking. As for what this ultimately means, I will now make a rather dour point. A fashionable explanation for the recession among both politicians and many “Main Street” types is that greed is what did us in. The greed of the bankers, the hedge funds, the fat cats, the small cats, whomever – greed is the culprit. But that doesn’t explain everything by a long shot. Even the greediest person doesn’t want the party to end and the money to stop coming in. Might it be possible that they weren’t able to ask the questions that might have led to certain debt instruments having never been created? Financial services employees have more information available to them than decision makers any other industry, and still here we find ourselves. Think about how many times each day similarly misinformed decisions are made inside corporations all across the world. The information is there, but we are more often than not letting it rot on the docks.

What makes the distinction between information and intelligence critical is that information is getting more accessible by the day. This democratization of BI is evidenced nowhere more so than at Microsoft. In 2008, Microsoft unveiled several projects in the late stages of development that it claims will put BI capabilities at the fingertips of more users than ever before. “Project Madison” will massively increase Microsoft’s information storage capabilities, while the “Kilimanjaro” and “Gemini” projects together will provide easy-to-use reporting and analysis tools designed to drastically reduce the complexity of using traditional BI tools – all at very low cost compared to large-scale ERP implementation. The possibilities abound. But I still ask the question: what are all of these newly empowered users going to do with all of this information once they can access it at the push of a button?

I am excited by the idea of so many more information workers being able to ask the questions that end up driving businesses to continuously reinvent and perfect themselves, but I worry about relevance. Will these capabilities actually increase the amount of intelligence available to decision makers? Any business decision can be thought of as a bet that some desired future state will materialize as a result of a present course of action. Business intelligence tools as we know them more often than not do not help us make more intelligent bets when it comes to the future. The problem is that we think they do. More data often makes the task of identifying the true predictors of business success and isolating their effects more difficult. In order for a company to get the most out of its data, it needs PI as well as BI capabilities at the fingertips of decision makers. For marketing and pricing to become a more fact-driven corporate discipline, we must recognize the need not for more data, but ways of evaluating the probability of outcomes based on only the factors that matter. This is not child’s play. Information alone, however well-groomed, is simply not sufficient to meet this need.