Search This Blog

Showing posts with label information overload. Show all posts
Showing posts with label information overload. Show all posts

Saturday, February 25, 2012

Searching for Red Stockings: The Myth of Information Overload


As the internet advocate Clay Shirky noted, everybody who talks about information overload starts with the graph with the telltale ascending line and the litany of the troubles it entails. As the line informs us, information is increasing exponentially, and we can barely deal with it intellectually and emotionally, or more and more often, we can’t. And the solution?  It is here that the rallying cries diverge.  

Scary Graph (from Basex.com)

On one side there is Shirky, who assigns the problem to filter failure, and why not? It’s a reasonable thing after all to suppose that if we had better ways to sort out information, we could cull the bad from the good, and be able to significantly reduce the information we have to cope with daily.  Search, social media, and e-commerce firms of course concur, and are rapidly improving their search algorithms (using of course information about you that you voluntarily or involuntarily port over to them) so  you can find what you need the first time.

On the other hand is the internet critic Nicolas Carr, who attributes information overload to filter success. In Carr’s opinion our filters are working all too well, and the problem is that they are getting better and better. 

Thus,
“….The real source of information overload…. is the stuff we like, the stuff we want. And as filters get better, that's exactly the stuff we get more of.
It's a mistake, in short, to assume that as filters improve they have the effect of reducing the information we have to look at. As today's filters improve, they expand the information we feel compelled to take notice of. Yes, they winnow out the uninteresting stuff (imperfectly), but they deliver a vastly greater supply of interesting stuff. And precisely because the information is of interest to us, we feel pressure to attend to it. As a result, our sense of overload increases.”

Implicit in both arguments is this premise:


The information we want is the same as the information we need.


This is an argument for the curing salve of better filters (to fine tune what we want, since our wants are finite) or a call for mass despair (because our wants are infinite, and thus overwhelm us when they are invariably served by the web). This premise derives from an assumption that in our hubris we are wont to make: that humans are rational agents that know what they want and why.

But what if this was not true?  What if we are at root irrational creatures who delude ourselves into thinking that we know what we want and why we want it? What if the information we want is more often than not different from the information we need? If this is true, then to paraphrase Shakespeare, our fate is not in the stars (or rather the cloud), but in ourselves, because if the information that we want is often not the same as the information we need, then we need to be aware how to distinguish our wants from our needs and how and when  to constrain the former.  In other words, for information overload, the key is to understand how our basic motivations work.

The question that Shirky and Carr beg is thus elemental: Why is information of interest to us, because it is important, or because of something else?  To answer this question, let us illustrate how a basic search was performed over the last few generations by going to our metaphorical sock drawer in search of red stockings.
It’s 1912, and you as t-shirt manufacturer want to begin a production run of commemorative t-shirts of the Boston Red Stockings triumph in the World Series. As soon as the game is over you receive an immediate telegraph of their victory, and it’s off to the races to start production.

It’s 1932, and you as a t- shirt manufacturer want to get started with your commemorative t-shirt run, and so you listen to the game on the radio, and upon its completion, get to work.

It’s 2012, and you as a t-shirt manufacturer want get to cracking on your production run celebrating the Boston Red Sox victory, and you follow the sox from college draft to preseason to all of their games through the World Series, and monitor all the social and news media who have something to say about it.

In all three time frames, the decision point happens in a second at a predetermined moment, namely when (hopefully) the sox win. The narrative of how that final fact (a sox victory in the final game) got there is irrelevant. No matter what era, the decision point is concise, precise, and momentary, and gets to you on time regardless of the media you use and irrespective of its background story.  There is no need to follow the narrative that describes the changing facts that get us to that point, as the point of the last man flying out in the last inning is all we need.

The difference between the three eras is that in the first era we could not follow the narrative that follows the sox on their way to the pennant, but in the latter era we could. But following the latter comes at a cost.  By following the progress of the sox we become diverted from other things of value, and suffer regret.  If these diversions are small scale and populate our working day, they become distractions and cause us to lose focus and attention. Finally, as we continually choose between distraction and staying on course, we become tense and nervous.

The metaphor of ‘information overload’ would seem to apply here, as every frame of every moment of the continuous narrative leading to the Red Sox pennant can and is considered by the sox fan. However, like a strip of static frames in a motion picture that give rise to a sense of movement or motion, the story is interesting because of the novel ways the narrative changes, and it is the changes that compel. Thus, although the ending is necessary for us to go about our business, the story that leads to it is compelling not because of what it is, but how it is continually transformed.

We can expand our simple Red Sox narrative to the narratives embedded in all the things we do that are being progressively revealed by the web. We need to know facts, but what obsesses us is the narrative or story that brings us to those facts.  The internet produces not just more information, but more narratives of information. We see not a picture, but a movie, not a note, but a score, not a phrase but a speech. Moreover, we conflate the importance of the narrative with the significance of its conclusion, or what we want with what we need. This is a dangerous delusion, for the stuff we want depends upon the narrative or facts in motion, but the stuff we need depends upon the facts sitting still.

We can get the facts we daily need in a half hour, but continually accessing the web to see a moving stock market, middle east crisis, or what Uncle Charlie is up to are never ending stories that excite us, engage us, but ultimately bring us down. A narrative is of course still important if our behavior necessarily changes in tandem. In this case the narrative is ‘feedback’. Thus, a quarterback’s performance is determined by feedback during the moment to moment course of the game. However, for the stadium audience, this feedback is entertainment, and for those who attend to the ever expanding narrative on the game itself, an unnecessary and harmful obsession.


The Myth of Information Overload

As a metaphor, information overload attributes the psychological effects of the internet to what information is rather than how it is arranged.  But humans are above all novelty-seeking creatures, and novelty is enhanced not in the facts but in the stories they tell. Because the explanation for how the web influences us psychologically is based on core assumptions on human motivation that are faulty, we proceed with our daily lives under a dangerous illusion abetted unfortunately by the perverse incentives of our media providers to keep us hanging onto the story when the conclusion is all we need. Whether or not we can escape this illusion and its dire consequences depends ultimately on not just a better story, but also a better explanation as to how our minds actually work.

Sources:
Finding a better story to describe the emerging fact that wanting and liking aren’t the same thing takes you to the seminal work on the topic performed by the neuropsychologist Kent Berridge on the topic, or my own narrative on Berridge’s work and what it means.  Hopefully both make for some interesting explanations.

Monday, July 25, 2011

Feedback Overload


Since the invention of writing, the written word has literally piled up. Indeed, from very early on, mankind has been overloaded with information. However, the problem posed by of information overload is not in a metaphorical stack of stuff, but in our relative inability of finding the needle of information we need in the haystack of information we don’t. Things like the Dewey decimal system, book indexes, and a helpful librarian barely addressed the problem until the invention of the internet search engine allows us to find our need, or in this case, needles. As the pundit Nicolas Carr opined[i], the problem we confront today is not finding a needle in an infinite informational haystack, but finding an infinite stack of needles that all merit consideration. Nowadays, when we electronically search for any topic, we are provided with many similar bits of information that allow us to more precisely fine tune or correct the deficiencies of knowledge. This error correction or feedback function represents a progressive resolution of the discrepancies between what we do and don’t know. Feedback may represent unexpected changes in our progress to a goal and/or unexpected changes in our knowledge of the nature of a goal. Feedback of course is essential to learning, but consequential to that learning is the increased activity of midbrain dopamine neurons, and it is the neuro-modulator dopamine that enables the consolidation of memory as well as heightened alertness and attention on the task at hand. But dopamine also increases positive affect that adds momentary value or ‘incentive salience’ to behavior, but does not intrinsically predict the overall or long term goodness or utility of behavior. Put a bit differently by the neuro-psychologist Kent Berridge, “The brain results suggest that pure decision utility—and not predicted utility—is raised by activating mesolimbic dopamine systems[ii].”  What this means is that the importance of the decision in the moment, or its ‘decision utility’ does not influence its long term or ‘predicted utility’. The implications of this are profound, for as the marginal utility of examining each informative ‘needle’ declines, the successive needles of information remain novel, and we continue to dwell on nearly redundant links of information not because they are useful but because they are new. In other words, whereas in the past impoverished feedback environments caused us to waste much time looking for information, the rich feedback environments heralded by improvements in web search lead us to waste much time looking at information! This means that we will be affectively and not rationally inclined to overstay our welcome on sites that not only provide us what we want and need, but infinite variations of the same information that we ‘want’ but don’t need. The problem thus is not information overload, but ‘feedback overload’, as the ever increasing amount and granularity of information feedback provides greater and greater detail that can increase the short term or moment to moment value of behavior to the detriment of our long term interests.



Information Search, CA 1960




Information Search, CA 2011

This increase in the momentary incentive salience of behavior can be used to conform with (if not predict) practical ends, but its ultimate value depends upon whose practical ends.  For example, the ‘Khan Academy’ (khanacademy.org) is an online math tutorial that uses rich feedback embodied in badges, scores, hints, etc. to increase the decision utility of performing math exercises in service of the predicted utility of long term mastery of say, the mathematical calculus. On the other hand, a Google search also provides rich feedback including social network feeds, instant messaging, videos, helpful links, and now badges in the service of the predicted utility of Google, namely advertising. 

Ultimately, the problem is not that we are lost in a haystack, but that we are proverbially resting on a bed of pins and needles with each pin needlessly diverting our attention. The notion of ‘feedback overload’ means we are neurologically inclined to overvalue the short term importance or salience of new information, and when new information scales in amount and availability, we begin to live for a moment that may not conform to our ultimate good. For the rich feedback mechanisms provided by the internet, whether it is social media of just plain search, the solution to this problem is not better filtering of information or better feedback (as this merely acerbates the problem), but less, and can only be accomplished by constraining what information you can see, or when you can see it. The simple solution is keeping your personal library and newspaper, and severely restricting your time with search tools (the internet) that work too well. As internet feedback trends to infinity in ever morphing detail and availability, this will be our only option to spare us a new dark age caused by being blinded by the light.

References:
Berridge, K. and Aldridge, J. W. Decision utility, the brain, and the pursuit of hedonic Goals, Social Cognition, Vol. 26, No. 5, 2008, pp. 621–646


Nicolas Carr, Roughtype (blog)



Saturday, June 11, 2011


Information Overload: The Anatomy of a Delusion


Sometimes the proper use of a product is ignored, owing not to a lack of awareness of its effectiveness, but to a lack of an explanation of how it works. For example, in 1839, draping your bed with finely woven gauze curtains was known to ward off malaria.  Malaria, which  means bad air, was generally considered to live up to its name, and be the literal result of well, bad air. Thus, hanging curtains around your bed presumably filtered the air, and thus helped prevent malaria. Nonetheless, going beyond mere curtain hanging to properly covering your bed with curtains didn’t catch on until a proper explanation of malaria was at hand that suggested as a matter of course the preventative measure of mosquito nets. But this didn’t stop another invention that stopped malaria just as well by treating it was thought, all that bad air. The invention was the inspiration of early 19th century Florida physician John Gorrie, who in an experiment sealed off a room and conditioned the air with a special device of his own invention. Of course, conditioning the air wasn’t the cause of malaria, but it had the incidental benefit of cooling it, and air conditioning lived on because it made you comfortable, and not because the system incidentally didn’t allow mosquitoes in. The explanation of mosquitoes as the carrier of malarial parasites resulted in the better use of available procedures that stopped mosquitoes, namely mosquito netting, and it provided the source of new procedures that stopped malaria (e.g. fumigation, swamp draining) as well as explaining why other procedures worked (e.g. air conditioning systems) and why other procedures of the day (e.g.handkerchiefs soaked in vinegar; garlic worn in shoes) that seemed to work, didn’t.







John Gorrie’s ‘Bad-Air’ Conditioner


Now consider a modern malady that doesn’t kill you, but manages quite well to kill your time. This is the modern bane of ‘information overload’. “Information overload refers to the difficulty a person can have understanding an issue and making decisions that can be caused by the presence of too much information” (Wikipedia). A logical problem with this definition is that we have always been in the presence of too much information, as a simple walk through any library can demonstrate. Before the internet, navigating this wealth of information was rudimentary and difficult. You used a card file to determine what you needed and walked around different book stacks to find it. Invariably what you found was often not exactly what you needed, but it had to do because the transaction cost for information, namely rifling through card files and roaming book stacks simply was too high.

In the internet age, our filters are immeasurably better, and we can get information tailored to fit our request, or employ intelligent agents that use a mere history of our internet behavior to find the information we need. Moreover, we can get this information for a negligible transaction cost, or for free. The problem is as any web search demonstrates is that we are handed with many variations of the same information, or information that is nearly redundant. In other words, different variations of the same information in different sites essentially restate the same information. For example, perform an internet search for ‘high gas prices’ and you will get a score of links to different articles that discuss high oil prices. If we were rational agents, we would read one or two articles and then cease, knowing that the marginal usefulness or utility or reading a third or fourth article decline markedly as they would restate the same information and generally arrive at the same conclusions. But the fact is we don’t. Indeed, we may read many more articles on the topic and even far removed from the topic, and then come back to the web after a few hours to read more. The same goes for any matter of internet searching, whether it is social media, email communications, or just looking up a sports score.

Ultimately, the problem of information overload derives from the implicit assumption as to what type of information is actually overloading us, and what overloading means. To gain our attention, information must not just represent random correlations between events, but rather correlations between events that have predicted value or utility.  But information also has a degree of novelty, and it is the integration of novelty and utility and not utility alone that determine the importance or ‘incentive salience’ of moment to moment behavior.  Utility and novelty are integral aspects of information, and even if information has marginal utility it still becomes an object of desire if that information is novel. Thus an individual may initially access his email because its utility far outweighs the novelty of discovery, but by the fortieth time in the day that he checks his email, the novelty of the act far outweighs its utility. Moreover even if we know the behavior has low utility, we often keep on searching. In addition, we are affectively primed to search for new information by in effect (or perhaps in affect) looking forward to accessing our email, social network, news feed, etc.

The concept of information overload implies that we are deciding between an abundance of information that is of uniform utility, but this is not true. Because of well honed filtering systems provided by the web, we can generally find what we want quickly. The problem is that the internet generates nearly redundant information that is distinguishable not by its usefulness, but by its novelty. In other words, we are not overloaded with information that is primarily useful, but are overloaded with information that is primarily novel. By generating infinite variations of the same information, the internet primarily generates not useful but novel information, and the overload is not in the information we want and need, but in information we want but don’t need.

So the problem, as the example of Gorrie and his air-conditioner demonstrates, is that ‘information overload’ like ‘bad air’ is a false cause for a very real problem, mainly the web as a source of distraction rather than value. So the real solution is not to create better filters, since they ultimately don’t matter, but rather to limit our access to the web to those times when it logically means something, not when it doesn’t. In other words, the solution is not found in better filters, but in merely draining the swamp.

Saturday, April 2, 2011

Information Overload


“Why are you fearful of the whole ocean swallowing you, when in fact you can drown in a cup of water.” Epictetus, 110 A. D.

We often moan that there’s not enough time in the day because there’s too much to do, but now the common complaint is that it’s because you’re doing too much. Information overload is the bugbear here, but it has been the bugbear since our ancestors were bugged by bears. Humankind has always been faced with more information than it can handle, but we learned to handle it by filtering. Like a chess master pondering the numberless moves that can sequentially secure checkmate, humans parse between information that is necessary, optional, or redundant. But they are also sensitive to novel information as well, and this ingredient can change the behavioral calculus in ways that make it impossible for us to out good information from bad.

Consider if you would, your uncle Charlie. It’s 1965, and living as he is in a faraway town, he’s always available to you, and is merely a phone call away. Unfortunately, long distance phone calls back then set you back twenty five cents a minute, so when you were calling Uncle



Social Networking Device, circa 1965

Charlie, it was sure to be about something important. Although infinite information about Uncle Charlie was available, the transaction cost of obtaining that information insured that the information you got from you uncle had a high predicted value, usefulness, or utility.

Now it’s 2010, and Uncle Charley is still around, along with his infinite experiences that he was always willing to share. The pay phone is long gone now, and Uncle Charley is now plugged into the entire electro-magnetic spectrum. And you can access his every move and every thought through myriad devices and services that provide you Uncle Charley, all the time. So whether it is Twitter, Facebook, Foursquare, instant message, email, or Skype accessed through your iphone, ipad, laptop, or even future permitting, cranial implant, Uncle Charley is no more than an eye blink away.

More important, Uncle Charley is now ‘free’, and you can access him with minimal cost or fuss. So even though the value of accessing Uncle Charley from moment to moment is near zero, we still end up accessing Uncle Charley, a lot. In fact, we are ‘overloaded’ with Uncle Charley as well as infinite minutiae of minimal utility but high urgency. In fact, as in Epictetus’ maxim, we find oceans of information in a few ounces of water but historically have not been drowned in information because access to information comes at a cost. Now as the cost approaches zero, attend we must, and end up drowning in cup.

When the cost of information trends to zero, so does its marginal or incremental utility. However, the affective value of novel information stays low but constant, and when the threshold is passed we end up valuing information not because it is valuable, but because it is new. Thus when information is dear, we value it because of its utility, but when it is cheap we value it because it is novel. But unlike rational goods, novel goods cannot be easily parsed or handled according to rules, hence we become ‘overloaded’ with them, and that is a problem even a computer can’t help us with.