What Happens When We Can’t Keep Up with Information

I ran into a former colleague the other day who, as it turns out, recently left his job and presently spends his days building options pricing models and trading from home on his own accounts. In turn, I described to him some of the recent work that we have done in revenue optimization and particularly the breakthroughs that we have engineered for processing data. His face scrunched up a bit, and his response was uncharacteristically blunt: “You can always process numbers quickly if you need to,” he smirked.

Not so, in fact. When you start asking extremely detailed questions that require combing through years of detailed historical data and then performing mathematical transformations on each of those figures, you will find out rather quickly the limits of processing speed when your results finish compiling in a week or so. The thing is that most of us never push up against the processing speed frontier. We can see that every year computers get faster, chips get smaller, and Excel seems to have more rows. Moore’s Law prevails. The trouble is that all the while the rate at which the data universe expands is screaming past advances in processing capabilities, and that rate does not fluctuate with the economic downturn. Consider the markets for microprocessors, which allow us to perform those calculations and manipulate data, and hard drives, which allow for storage of data. Microprocessor sales have been dealt a sharp blow by the global downturn as computer sales have slowed, but worldwide shipments of hard disk drives (HDDs) roughly maintained 2007 levels even in the worst quarters of the recession (and the drives themselves contain more memory). Solid state and flash memory shipments were down, but the evidence suggests that this is due to consumers substituting HDDs for other types of memory, rather than simply not storing more information. The demand for data storage, while not completely recession-proof, is nonetheless of the hardier variety.

Simply put, information of all kinds accumulates faster than we can analyze it. We are losing the race, and the gap is widening, not shrinking. As for what this ultimately means, I will now make a rather dour point. A fashionable explanation for the recession among both politicians and many “Main Street” types is that greed is what did us in. The greed of the bankers, the hedge funds, the fat cats, the small cats, whomever – greed is the culprit. But that doesn’t explain everything by a long shot. Even the greediest person doesn’t want the party to end and the money to stop coming in. Might it be possible that they weren’t able to ask the questions that might have led to certain debt instruments having never been created? Financial services employees have more information available to them than decision makers any other industry, and still here we find ourselves. Think about how many times each day similarly misinformed decisions are made inside corporations all across the world. The information is there, but we are more often than not letting it rot on the docks.

Subscribe   |