I have implied this in other entries in this blog, but I might as well say it outright: The library and information science profession needs to come to terms with the issue of standards for (i.e., rules of) evidence for performance, statistical, and advocacy research data. There, now I’ve said it.
I recently read the short and enjoyable book Graphic Discovery: A Trout in the Milk and Other Visual Adventures by statistician Howard Wainer.1 The subtitle of the book comes from something Henry David Thoreau wrote. During a dairy strike in 1850 in New England people began to suspect that dairy owners were watering down the milk supply. This led Thoreau to write this entry in his journal:
Sometimes circumstantial evidence can be quite convincing; like when you find a trout in the milk.2
Wainer’s main point, one certainly made also by others like William Cleveland and Edward Tufte, is that well designed graphical representations are invaluable for exploring and understanding data. Graphical representation of data can lead to revelations about data and the underlying phenomena they describe that would otherwise be missed.
But, alas, Wainer and the others warn that the design of graphs can serve to mislead readers. Statistics can lie. So, you have to figure that statistical graphs might fail a few polygraph tests, too. Now re-sensitized to this possibility, I am looking closer at graphs I encounter. A graph appearing in an article in the Nov. 9, 2009 issue of Business Week was easy prey for my renewed vigilance.
Click on this image to see the Nov. 9 Business Week graph.
Unfortunately, the electronic versions of this article available from EBSCO, LexisNexis, and other databases omit graphics altogether—an aggravating defect of digitization, indeed. To see the graph, click on the image above.
In the Business Week article, “The GDP Mirage,” author Michael Mandel argues that the economic index, Gross Domestic Product (GDP), is incomplete because it does not measure “intangible investments” corporations make. By overlooking these investments, Mandel claims, the U.S. is “navigating…with fragmentary information” (p. 36). You can get the gist of Mandel’s ideas from the article itself. For now, I just want to point out that the aim of the graphic is to illustrate the author’s argument.
Notice that the Business Week graphic consists of three charts. Rather than having an individual title for each chart, a caption at the top forms three surrogate titles:
Reported GDP jumps ahead of jobs [left graph]…but the GDP stats don’t count R&D cuts [center graph]…or lost jobs for knowledge workers [right graph].
The implication is that if the GDP were to include statistics reflecting cuts to research and development and lost jobs, it would be a more valid measure of economic output. (The article doesn’t actually recommend that job loss statistics be included in revised GDP calculations, but we can ignore this inconsistency for our present purposes.)
This graphic has a numbers problem quite distinct from the GDP measurement challenge that concerns Mandel. The problem with the graphic is this: Two of the three charts report (let’s call these) actual data while the third does not. The left and right charts present data obtained from the U.S. Bureau of Labor Statistics, which we can presume were collected using accepted sampling methods. However, the center chart is—depending on how you look at it—either a convenience sample or merely a collection of anecdotes.
The center chart’s heading, “Selected Companies that Have Cut R&D Spending Over the Last Year,” suggests that the selection is some type of nonprobability sample. As seen in the chart, cuts for these companies range from roughly 12% to 36%. Nowhere, though, does the chart or the article tell us how the companies were selected or to what extent the percentages pertain to the larger set of U.S. corporations of interest.
What we have is anecdotal information masquerading as data! Even though the chart title is clear,3 placing the chart in the middle of two other charts that contain actual data is deceptive. The middle chart appears to be on a par with the other charts when it really is not. The center chart is mostly conjecture, the other two have firmer grounding.
Since the units of measure in that chart are percentages, the population parameters (in this case, percentages of decrease in R&D spending among all U.S. corporations of interest) are likely to be within some reasonable range, probably not ridiculously far from the range seen here.
But this is not the point. The author does not have any conclusive evidence about what this range actually is and he, or the creator of the charts, ought to say so. This is a case of pretending to have data that you don’t, in fact, have. This is, in other words, navigating with fragmentary information.
Wainer would not be so forgiving. He would call the center chart “nondata” since that is what it is. He also makes this wonderfully apropos pronouncement:
The plural of anecdote is not data.4
Sure, for particular purposes, quick-and-dirty selections and pseudo-samples can be justified. But, they do not deserve to be graphed. So, if you will permit me, I want to experiment with a possible contribution to the set of standards for evaluating evidence that the library and information science profession might someday establish:
Standard XV.1.c. Since anecdotal information represents only itself, it shall not be portrayed, nor presented graphically, in a way that implies that it describes any phenomena in the aggregate.
Okay, so I can’t think of very good wording. Thankfully, there’s plenty of time for re-working that sentence…
1 Wainer, H., 2005, Graphic Discovery: A Trout in the Milk and Other Visual Adventures, Princeton, NJ: Princeton University Press, 2005. 2 Quoted in Wainer, H., p. 81.
3 I don’t mean to say that the chart is clearly titled, but that, once you are able to find it, the title (or is it a subtitle?) has an unambiguous meaning.
4 Wainer, H. p. 57.