Pardon my “I told you so” attitude, but Bell does underscore the message I harp on in my February 16 and January 5 posts. Still, I don’t agree with his more general conclusion, that technology has negligible effects on our lives. Of course, Bell doesn’t say exactly that. He is talking about demonstrable effects of technology (defined as video game and Internet use) on our neurons and behavior. Besides, Bell’s conclusion would not apply if we had research questions about possible effects of technology from sociological, political, cultural or economic perspectives.

Speaking of sociological impacts of technology, I want to recount a naturalistic participant-observer field study I accidentally conducted on this very topic! Well…okay, I admit this wasn’t exactly field research. It is just a story I offer because I think it is interesting and fairly revealing.

Last month while in San Francisco I visited the Internet Archive. I wanted to see the Espresso Book Machine they had there, one of I think only nine in existence. I had watched the video of this amazing contraption, but wanted to see, hear, smell, and feel it in real, four-dimensional space/time.

Arriving at the Internet Archive’s Funston Street location, my friend and I found the main entrance of the neoclassical building inaccessible due to renovation. Around the side of the building we saw an art deco brass door propped slightly open at the bottom of some steps. To the left was a glass-enclosed annex with the large sign, “READING ROOM.” My immediate hunch was that this was a con-verted Christian Science Reading Room. But—and this is part of my participant-observation study—I noticed in myself a willingness to suspend quick interpre- tations of sign(al)s like this. “The Internet Archive is about reading. That could well be it’s own logo, ” I thought.

As my friend and I approached the entrance we noticed the gold-lettered greeting centered on the glass door: “ALL ARE WELCOME.” Hmmm. Christian Science and the Internet Archive do share things in common: They are both movements inspired by prophecy, the former from divine sources, the latter from secular, if more evangelistic, ones. And both are committed to open access to materials. So, I was perfectly willing to be open-minded about whose openness we were enjoying.

Walking in, we saw an unattended retail counter with two or three glass shelves displaying various books, none of which were by Mary Baker Eddy, I noted. Behind the counter were several messy stacks of papers. In an adjacent room we could see people sitting studiously at workstations. Two small signs were posted outside the entrance to the room: “Visit OpenLibrary.org” and “Please Wipe Your Feet.” We could feel the openness virtually expanding as we stood there. We wiped our feet and walked in. A person who I believed to be an Open Library patron stood up, greeted us, and asked what we wanted. His body language, as well as a closer look at the workstations, told us that we had overstepped some very invisible boundary.

Damn! We had misread every cue! This was not an open library. It was a digitization project work room, one that was—how shall I say—PRIVATE. RESTRICTED. EXCLUSIVE. I guessed that our obligatory greeter was a shift supervisor. He was not particularly skilled in the niceties of public relations, although, to be fair, he was probably distracted by a deadline. I apologized for our truly ignorant intrusion and explained that I had come to see the Espresso Book Machine. He said he had never heard of it and was unable to guide us to any further information about it.

Leaving the Reading Room, we saw a couple of people exiting the door we had passed by. So, we gave that entrance a try. Stepping in, we were again in a common production area of some sort. Fortunately, a person standing inside the door was quite helpful, once I explained my quest—to observe the Espresso Book Machine at work.

“Oh, I’m sorry. That machine isn’t available for public viewing. It isn’t in operation. Come to think of it, it never did work right. I mean, after being serviced it would run for maybe four hours and break down again. And it had problems trimming the books right. See, here’s one.” He picked up a paperback nearby and showed us where the delerious machine had gnawed into the body of the text. “So we have it stored away. It was machine number 3 [or 4 maybe he said ].” I avoided the temptation of asking whether, during its shutdown sequence, the machine began humming “Bicycle Built for Two.” Nor was I so insensitive as to ask how much they paid for the thing.

Our host also recounted one of the missions of the Internet Archive. “Our founder has studied how expensive libraries are to operate. Such a large physical inventory to maintain and so laborious. But, it is possible to provide open access to books much more cheaply in an on-demand mode. Books can be delivered just to those who need them when they need them.” He went on to describe his institution’s ambitious project to catalog the whereabouts and formats of all available electronic books.

As our conversation drew to a close, he handed us each a small souvenir Archive-published, copyright-free paperback. Mine was Edward Lear’s 1889 Nonsense Drolleries: The Owl & The Pussy-Cat and The Duck & The Kangaroo. We thanked him for his graciousness and left.

This story, then, is about access to information, namely, my “seeking behavior” and how I eventually got my “information need” met (sort of). The story touches on other issues, some of great import, like the follies of automation and the possibility of library extinction; and on sticky policy issues, like the total cost of technology ownership and copyright confusion. It can also be viewed as a mini-allegory about drawing conclusions from evidence and the effects of our perceptual and conceptual biases (like gullibility). Literally, of course, the story is about the mundane idea of proper signage.

I am concerned, though, with how engagement with techno- logy changes our social mores in everyday circumstances. How natural it has become for cell-phone users to deny the existence of other human beings standing just a couple of feet away. And how the Internet Archive has not felt pressed to acknowledge any body’s physical presence by maintaining a public entrance. Probably when the Archive’s building renovation is completed public physical access will improve, remedied by those pioneers of usability, architects. Nevertheless, I have come to believe that technology organizations ultimately take on the characteristics of the products they hawk. Access is free, open, and unfettered as long as you follow sanctioned protocols.

——————————-

* “Open, sesame” is reported to be the anglicized version of the Arabic “Open, simsim.” See Wikipedia’s entry on Ali Baba

I recently read the short and enjoyable book Graphic Discovery: A Trout in the Milk and Other Visual Adventures by statistician Howard Wainer (Princeton, NJ: Princeton University Press, 2005). The subtitle of the book comes from something Henry David Thoreau wrote. During a dairy strike in 1850 in New England people began to suspect that dairy owners were watering down the milk supply. This led Thoreau to write in his journal, “Sometimes circumstantial evidence can be quite convincing; like when you find a trout in the milk” (quoted in Wainer, p. 81).

Wainer’s main point, one certainly made also by others like John Tukey and Edward Tufte, is that well designed graphical representations are invaluable for exploring and understanding data. Graphical data presentation can lead to revel- ations about data, and the underlying phenomena they describe, that would otherwise be missed.

But, alas, Wainer and the others warn that the design of graphs can serve to mislead readers. (Statistics can lie. So, you have to figure that statistical graphs might fail a few polygraph tests, too.)  Now re-sensitized to this possibility, I am—at least in the short term—looking closer at graphs I encounter.  A graph appearing in an article in the Nov. 9, 2009 issue of Business Week was easy

                                     Click on this image to see the Nov. 9 Business Week graph.

prey for my renewed vigilance. Unfortunately, the electronic versions of this article available from EBSCO, LexisNexis, and other databases omit graphics altogether—an aggravating defect of digitization, indeed. To see the graph, click on the image above.

In the Business Week article, “The GDP Mirage,” author Michael Mandel argues that the economic index, Gross Domestic Product (GDP), is incomplete because it does not measure “intangible investments” corporations make. By overlooking these investments, Mandel claims, the U.S. is “navigating…with fragmentary information” (p. 36). The reader can get the gist of Mandel’s ideas from the article itself. For now, I just want to point out that the aim of the graphic is to illustrate the author’s argument.

Notice that the Business Week graphic consists of three charts. Rather than having an individual title for each chart, a caption at the top forms three surrogate titles: “Reported GDP jumps ahead of jobs [left graph]…but the GDP stats don’t count R&D cuts [center graph]…or lost jobs for knowledge workers [right graph].”  The implication is that if the GDP were to include statistics reflecting cuts to research and development and lost jobs, it would be a more valid measure of economic output. (The article doesn’t actually recommend that job loss statistics be included in revised GDP calculations, but we can ignore this inconsistency for our present purposes.)

Its own title notwithstanding, this graphic has a “numbers problem” quite distinct from the GDP measurement challenge that concerns Mandel. The problem with the graphic is this: Two of the three charts report (let’s call these) “actual data” while the third does not. The left and right charts present data obtained from the U.S. Bureau of Labor Statistics, which we can presume were collected using accepted sampling methods. However, the center chart is—depending on how you look at it—either a convenience sample or merely a collection of anecdotes.

The center chart’s heading, “Selected Companies that Have Cut R&D Spending Over the Last Year,” suggests that the selection is some type of non-probability sample. As seen in the chart, cuts for these companies range from roughly 12% to 36%. Nowhere, though, does the chart or the article tell us how the companies were selected or to what extent the percentages pertain to the larger set of U.S. corporations of interest.

What we have is anecdotal information masquerading as data! Even though the chart title is clear,* placing that chart in the middle of two other charts that contain actual data is deceptive. Due to its location and bar-graph style, this chart appears to be on a par with the other charts when it really is not. The center chart is mostly conjecture, the other two have firmer grounding.

Since the units of measure in that chart are percentages, the population parameters (in this case, percentages of decrease in R&D spending among all U.S. corporations of interest) are likely to be within some reasonable range, probably not ridiculously far from the range seen here.

But this is not the point. The author does not have any conclusive evidence about what this range actually is and he, or the creator of the charts, ought to say so. This is a case of pretending to have data that you don’t, in fact, have. Or, in the Mandel’s words, navigating with fragmentary information. Wainer would not be so forgiving; he would call the center chart “nondata” since that is what it is (p. 57). On that same page Wainer also makes this wonderfully apropos pronouncement:

“The plural of anecdote is not data.”

Sure, for particular purposes, quick-and-dirty selections and pseudo-samples can be justified. But, they do not deserve to be graphed. So, if you will permit me, I want to experiment with a possible contribution to the set of standards for evaluating evidence that the library and information science profession might someday establish:

Standard XV.1.c.    Since anecdotal information represents only itself, it shall not be portrayed, nor presented graphically, in a way that implies that it describes any phenomena in the aggregate.

Okay, so I can’t think of very good wording. Thankfully, there’s plenty of time for re-working that sentence…

—————————————————
* I don’t mean to say that the chart is clearly titled, but that, once you are able to find it, the title (or is it a subtitle?) has an unambiguous meaning.

Compared to standards of good graphing practice that Howard Wainer, William Cleveland (The Elements of Graphing Data, Murray Hill, NJ: AT&T Laboratories, 1994) and others promote, the Business Week graphs are pretty damned bad! The axis labels are too difficult to find, first, because the charts are overpowered by thick, all-black bars and bold-fonted category labels (company names and occupation categories). And, second, due to small fonts, crowding, and misplacement.

In the left chart, the label “Percent” has the wrong orientation since it apples to the vertical axis. The chart’s horizontal axis has no label. Thanks to the chart designer’s use of Roman numerals we can guess that the units must be quarters on an annual economic calendar. Squeezing the legend into the data portion of this chart violates a cardinal graphing design principle: Don’t let clutter make the data more difficult to see. Though less important, the word “Forecast,” a note for the single GDP data point at quarter III of 2009, appears in a larger font than both axis labels and tickmark values. Not good.

In the center and right charts, the labels, “Percent Change in R&D Spending” and “Percent Change in Employment,” are misplaced. Both should appear on the lower horizontal axes near the the appropriate grid marks. Both labels include asterisked notes that imply the labels are meant to serve a dual purpose as titles (or subtitles). This confusion could be alleviated by creating descriptive chart titles that include the notes information (no need to separate it), and then inserting fully descriptive labels adjacent to the axes.

Grade this graphic earns:  D-

There is no branch of library economy more important, or so little understood by a librarian as helps to himself, as the daily statistics which he can preserve of the growth, loss, and use (both in extent and character) of the collection under his care. The librarian who watches these things closely, and records them, always understands what he is about, and what he accomplishes or fails to accomplish. The patrons to whom he present these statistics will comprehend better the machinery of the library, and be more indulgent toward its defects.     Public Libraries in the United States of America, Warren, S.R. And Clark, S. N., Eds., Washington, DC: U.S. Bureau of Education, 1876, p. 714.

Interesting that use of library statistics for advocacy purposes was recognized in 1876.

Early in the twentieth century our current ideas about performance measurement were already well understood, long before the practices of program evaluation, evidence-based management, and performance scorecarding were formalized. Arthur Bostwick, late director of St. Louis Public Library and Librarian of Brooklyn Public Library, wrote this in his book published in 1917:

No business can be properly carried on without a system of accounts. These may involve only money received and expended, but they may and should extend much further. The collection and tabulation of such [financial and performance] data have come to be regarded as indispensable by shrewd businessmen; and large corporations do not hesitate to spend considerable sums in employing a force of experts and clerks especially to gather data of this kind and to tell what they mean…    The American Public Library, Bostwick, A.E., New York: D. Appleton & Co., 1917, p. 253.

Bostwick also dealt with the ideas of accountability, continuous improvement, careful analysis of library statistics, and a tiered approach to evaluation data-collection:

Information of this [financial and performance] kind is gathered with either or both of two different purposes in view—to satisfy the legitimate curiosity of the person managing the business, or of some one who has a right to know how it is going on, whether it is succeeding or failing and just what it is accomplishing; and, secondly, to furnish a basis for improvements or changes, to indicate weak points and points of strength, so that the business may be reënforced along the former and extended along the latter.

…If the latter [purpose is intended], a more detailed and analytical study is made of the data, which are compared and tested in all possible ways to reveal unsuspected facts. When something is thus brought to light that seems to call for further investigation, additional data are collected. (p. 254)

He also preached about the imprecision of library statistics, a topic conveniently overlooked in our profession nowadays (as is the importance of assuring the validity of data presented in library advocacy reports also):

It should not be forgotten, either by those who collect and report these statistics, or by those who read them or use them, that they are of various degrees of exactness…In any kind of scientific measurement the limits of probable error are always mentioned to give an idea of the degree of accuracy. The less the probable error, the greater the accuracy. It is never stated that there can be no error and that the accuracy is exact; this would be simply ridiculous. The same holds good in library statistics. In the average report nothing at all is said of accuracy; the reader is left to conclude that all the data are exact, or at least that there is no difference in their degree of exactness. (p. 262)

Finally, the level of interest in this topic among Bostwick’s 20th century peers strikes a familiar chord:

But how much intelligent study of library statistics goes on in librarian’s offices, and how much modification or improvement in library methods and material results from such study, is something that we shall never now. It appears to be certain, however, that large numbers of librarians…look upon their statistics in the light of a necessary evil. They must be collected, because some thing of the kind is expected in the annual report, but they should be minimized, and, once in print, they should be dismissed from the mind. This attitude reminds one of the rural workman who used a dull saw because the amount of work before him gave him no time to stop and sharpen it… (p. 255)

In library advocacy research, though, message distortion is not amusing. I noticed a serious instance of this in a recent IMLS Research Brief which cites an American Library Association (ALA) report finding that patron use of library computers for job-seeking purposes has “greatly increased.” The ALA report is Job-Seeking in U.S. Public Libraries and the statement the IMLS brief cites is this:

“As part of site visits to public libraries in nine states [conducted in three annual studies], the study research team has found greatly increased use of library technology for job-seeking and e-govern-ment.” Job-Seeking in U.S. Public Libraries, American Library Association, Oct. 2009, p. 2.
(The emphasis in red is mine).

The ALA report is one of a series of “issue briefs” published by the Office for Research and Statistics that summarize and supplement key results from the multi-year Public Library Funding and Technology Access study. To date this project has issued three annual reports beginning with the 2006/2007 edition. (The project is a collaborative effort connected with the Public Libraries and the Internet longitudinal studies which began in 1994.)

Anyway, I wondered how big this increase actually was and what the level of job-seeking computer use had been before the big increase happened. So I went searching for the numbers in the ALA Public Library Funding annual studies. Turns out none of the studies measured frequency of job-seeking or e-government computer use by patrons. Nor did the studies compare frequencies of any reported computer uses from year to year. The 2006/2007 and 2007/2008 editions merely state that job-seeking and e-government were common uses reported by some patrons, without mentioning increases of any sort. The 2008/2009 edition reports increased patron job-seeking computer use, but does not describe this increase as substantial or “great.”

The quotation from the issue brief (above) says that the researchers detected this increase by means of interviews with staff and patrons during library site visits. These interviews, conducted in a few selected U.S. states each year, included a simple open-ended question to patrons: “What do you use [the library's computers] for?” Job-seeking and e-government made the lists of most frequent responses (and each state’s list apparently differed from the others). But, no frequency counts for these uses show up in the three studies, perhaps because the counts weren’t collected. (Site visits, as well as focus groups which the ALA studies held, belong to the category of qualitative research methods. As the ALA project illustrates, collecting essentially quantitative information using qualitative methods can lead to problems.)

Even if the researchers did tally these uses during the interviews, neither the interviewees nor the states where interviews were conducted were randomly selected. So, we couldn’t say the tallies represent the larger patron population nationally. The 2007/2008 study reports a convenience sample of about 200 patrons who were using library computers at the time (Libraries Connect Communities, American Library Association, 2008, p. 128). (See the NOTE below.) The same study then reports that “Interviews with users confirmed staff observations that much computer use in libraries is job-related…” (p. 131; I added the emphasis). Exactly how much they don’t say. And how often unrepresented patrons—say teenagers who had yet to show up after school—might use computers for job-seeking we can’t tell. (The 2006/2007 ALA study cites a 2006 Baltimore, MD study where library computer use was found to depend on age group. See footnote on p. 169 of the ALA 2006/2007 study.)

Selection bias of a different sort confounds year to year comparisons from the ALA studies. Because the annual interviews were conducted in different states where the job markets and online government services could differ significantly, there are no reliable baselines for comparing job-seeking and e-government computer usage between years. For instance, the 2006/2007 study included a site visit to a library in Nevada where staff reported long lines of patrons using library computers to apply for jobs at a newly opened gambling casino. Relying on such atypically high usage as a baseline for job-seeking computer use could mask actual increases in later study years in different states.

The question remains, what data are these “great increases” based on? None that I can find in the ALA studies. The issue brief does cite other figures that did increase over time, but these figures don’t describe patterns of computer use. The figures are from the survey portion of ALA’s studies, from questionnaire items that elicit library staff opinions. Staff were asked to identify the top five public Internet services that they believed to be “the most critical to the role of the library branch in its local community.” In the 2006/2007 study 44% of responding staff chose provision of job-seeking services as one of their top five priorities. In the 2007/2008 study this proportion was 62.2%, and in 2008/2009 it was 65.9%.

Basically, about 22% more votes by staff (each staff respondent got up to 5 votes) went to patron job-seeking services in the 2008/2009 survey than in the 2006/2007 one. While these vote tallies may reflect changes in staff perceptions of computer use during this period, the votes are only opinions, and don’t indicate how patrons actually used computers in libraries.

The truth is that we don’t have the usage data needed to support the assertion made in the ALA issue brief. Without valid baseline data, we can’t measure increases in patron job-seeking or e-government computer use at all, and we certainly can’t tell whether or not any increases have been great.

———————————-

NOTE:    Convenience sampling is a type of ‘nonprobability sampling.’ With nonprobability sampling, we have no statistical basis for claiming that our study findings describe the larger population that we had hoped our research would apply to. Using nonprobality sampling invites biased information into study results.

In a Library Journal issue later that year American librarian Frances Rathbone clarified the East Orange, NJ registration figures and suggested a modified statistic that might help in comparisons with U.K. data. Still, that statistic put East Orange registered borrowers at 33% of the town’s residents, causing Rathbone to write:

“It is an unusually large portion of the population, but our conditions are unusual. East Orange..[has] street after street of thrifty and well-to-do residences. There is scarcely a person to whom the book is not part of his life…It is surrounded by other cities, not open country, and whatever persons are in jails or workhouses are looked after by the county elsewhere.”
(Library Journal, Vol 33, No 12, Dec. 1908, p. 506)

More than a hundred years later unusual libraries with unusual data are still evident in U.S. public library statistics. The chart below (borrowed from my April 2009 post) shows how national library statistics are spread unevenly—or, in statistical jargon, have “skewed distributions.”*   Most libraries’ statistics fall within a given range while a few libraries’ data are especially high.

Distribution of Circulation per Capita for LJ Index $400K Peer Group

Yet, as Frances Rathbone explained, statistics that appear unusual are not necessarily wrong. There can be quite legitimate reasons for high figures. A good example is how libraries serving substantial numbers of patrons outside their legal service areas end up with very high per capita statistics.

On a continuum of library evaluation tools, library ratings are among the simplest and most rudimentary. Second-guessing, changing, or eliminating unusually high (or low) library statistics is inappropriate when calculating library ratings. Are a given library’s operations superb (or substandard)? Is the community an especially fertile (or infertile) environment for library services? Is the library highly attuned (or insensitive) to community information needs? Have data collection methods exaggerated (or under-reported) the data? Only intensive surveys and onsite audits of U.S. libraries would lead to the information needed to answer these questions.

—————————–

*   This is true for traditional measures like visits and circulation and also for newer ones like electronic usage.

I can’t vouch for the evaluation process used in the Best Small Library awards. But I do know that LJ Index ratings are no basis for drawing specific conclusions about the performance of the director or the library. Perhaps their performances have been exemplary, or perhaps unsatisfactory. It seems that LJ Index Star status would support the director’s side of this painful debate. Nevertheless, at the risk of sounding like a broken mp3 file, I have to repeat the caveat that my colleague Keith Curry Lance and I stress in our articles: Library ratings do not measure organizational effectiveness, performance excellence, service quality, or the extent to which a library meets community needs. This troubled library and board will need to look way beyond library ratings if performance is one of the issues they are grappling with.

And, since I’m preaching about how library ratings should be interpreted, I might as well share a more subtle and surprising fact with you: Ratings cannot provide much detail at all about an individual library. Sounds like a contradiction, doesn’t it? Giving a library an exact rating score, differentiating between low and high scoring libraries, and then saying ratings lack specificity? Because library ratings are based on aggregating data that are quite general, they are only useful for making (let’s call these) low-resolution statements about libraries. They give the general outline and shape of things from a given perspective, not a complete, filled-in, 360-degree virtual picture. In other words, as a general rule ratings are very general rulers.

Here’s a different way to think about this (if you will forgive the unpleasant connotations of this example): Consider psychological or psychiatric assessments of criminals. These assessments do not actually measure criminals on an individual basis. That is, the results of these assessments do not predict whether a given individual will or will not be a repeat offender. Instead, the assessments assign an individual to a certain classification or profile, and then describe how criminals matching this profile typically behave. Forensic psychiatrists and psychologists are careful to declare that their assessments do not absolutely describe the individual being examined. Rather, the results are primarily statements of probability based on group profiles.

Roughly speaking, a similar but much less rigorous process applies to the happier world of library ratings. Although the three- and four-digit scores appear precise, they don’t really “zero in on” a given library. Their only purpose is to assign libraries into final groupings. Ratings make very general statements about how libraries compare to their peers. But they do not provide details about how specific libraries under a given management team perform in their unique communities.

So, why consider ratings at all if they are such non-specific measures? Because they can fulfill an important role in promoting both libraries and library evaluation. Most importantly, they remind us of the need for libraries to be accountable to community stakeholders and funders. In this respect ratings represent a unique evaluation method that William Gormley and David Weimer examine in their 1999 book Organizational Report Cards. These researchers acknowledge that ratings and report cards are imperfect and “face all the major methodological problems encountered in designing performance monitoring systems and program evaluation” (p. 6).  

Yet, these problems do not outweigh the importance of having publicly available data about libraries as a catalyst for ongoing local evaluation. By measuring one dimension of performance—service provision—the LJ Index recognizes exemplary libraries while also encouraging all libraries to identify those assessment questions worth pursuing in high-resolution and full living color.

Respondents in the study were divided into four groups according to how supportive of public library funding they were. Somehow—the OCLC researchers don’t say how—certain questionnaire responses qualified respondents for assignment to each group. But the results don’t always make sense. For instance, 20% of “super-supporter” group said they were not definitely committed to voting in favor of libraries. And 6% of this group was either unsure how they’d vote or said they would vote “No.” Wondering how such uncommitted respondents ended up assigned to the group that is “super” supportive of libraries, I contacted the OCLC researchers. Unfortunately, I never did get a response to this question.

There are other perplexing aspects of the study, like details of the study sample: What was the sampling frame? Were subjects randomly selected? Is the sample geographically representative? Apparently, OCLC’s research is proprietary and not subject to the type of outside verification and review that is the mainstay of academic and scientific research.

So, now to a couple of things clearly evident from the 212-page report. The researchers devised two statistical indexes to describe key attributes of their study respondents: one based on respondents’ library visit frequency, the other on responses to specific questionnaire items. Imaginative charts comparing the two indexes for different respondent groups are spread throughout the first half of the report. An article I wrote examining these indexes appears in the current issue of Public Library Quarterly (PLQ), vol. 28 no. 3 (July-September) 2009.

In the PLQ article I point out that the indexes happen to have no units of measure. In the language of mathematicians, they are “dimensionless.” This makes it really difficult to interpret the repeated comparisons that appear in the study. This annotated graphic from the PLQ article attempts to explain a sample chart from the OCLC report:

                                 ©  2009  Taylor and Francie Group

I won’t elaborate on this graphic because I want you to read the complete PLQ article. Suffice it to say that my annotations are the rectangular bubbles, arrows, and parenthetical axis labels. I conclude that, while the indexes appear to be accurate,*   they are more complicated than need be, given that they are based on such simple data. As a result, they aren’t particularly useful.

My main message from the article is, though, that as consumers of advocacy research we need to be research literate, meaning that we have the knowledge necessary for examining research findings to determine whether study data and methodology really do justify the study findings. Certainly, we need our leading library organizations to pursue advocacy research projects like this one. At the same time, we have to look beyond deluxe graphics and polished text and focus on the actual substance of library studies. As information specialists and brokers, we have a special obligation to verify that research done on behalf of libraries is understandable, accurate, objective, transparent and relevant.

* OCLC declined to respond to my repeated requests for information necessary for auditing the data in their report. They did give permission for reproducing their charts in PLQ.

When I disapproved of WebJunction’s online competencies assessment questionnaire (see my April 22, 2009 entry), the WebJunction staff explained to me that the true objective for their surveys was to increase awareness of these competencies. I immediately wondered, “Well, how then will WebJunction measure awareness?”  But that is quite an irrelevant question when these questionnaires are actually teaching tools, not measurement instruments. Since the instruments don’t really have to measure anything, we don’t have to obsess about how reliable or valid they are. They can be evaluated (I guess) according to how well they apply proven methods for facilitating adult learning.

The irony of using a research instrument like a survey questionnaire this way will probably escape the majority of librarians (i.e. those who disliked library school research methods class.)  But here’s the story: One of the giant problems in designing behavioral science measures is making sure the measures don’t alter the thing you’re trying to measure. Measures are supposed to be unobtrusive. You would never trust a thermometer if you found that, while measuring the temperature of water, the thermometer also happened to heat the water! The same goes for questionnaires and tests in behavioral science and education.

Worries like this are old hat nowadays. Forget the antiseptic, hands-off approach. So easy and cheap to post online, the new questionnaires are designed to induce change by informing, educating, and motivating respondents. I ran across another one of these in connection with a new initiative on “21st century skills” launched last week by the Institute of Museum and Library Services (IMLS). This campaign presents a thoroughly modern take on the mission of libraries and museums. You can read the details and access the “self-assessment tool” here.

Still stuck in my 20th century research methodology paradigm, I found the IMLS questionnaire technically interesting. It is what I call a “Goldilocks instrument” since it uses a 3-point ordinal scale that amounts to a little, a medium amount, and a lot. The response options are something like this:

1. The institution rarely practices such-and-such 21st century skills enhancement task or technique
2. The institution practices the task or technique fairly often, or
3. The institution almost always practices the task or technique.

In several questions in the survey, this tripartite scale appears as less than 25% of the time, 25% to 75% of the time, and over 75% of the time. But you get the idea—small, medium, large.

Specific questionnaire items address a series of general institutional dimensions like accountability, leadership, partnerships, and so on.  (See the self-assessment tool matrix.)  Then, in each area, the institution is rated as being in one of three developmental stages:  Early, Transitional, or 21st Century. An institution’s Goldilocks responses fall conveniently into these stages (surprise!!).  If you perform a 21st century skill enhancement task less than 25% of the time, you are in the Early (Neolithic?) stage on that one.  If you perform it more than 75% of the time, you are thoroughly modern!

At the completion of the questionnaire, the self-assessment tool simply parrots back an institution’s responses in graphical form. There are “Recommendations” buttons users can click on, but the advice offered is pretty much the same, regardless of an institution’s rating: Use the results “to initiate a dialogue with your institution’s leaders, board, colleagues, and other stakeholders” so you can improve your rating. In Goldilocks measurement terms, having the most 21st century skills possible is always “just right!”

Obviously, the survey is a teaching tool, not an assessment. That’s why there is no need for the instrument to gauge how libraries and museums compare to any independently derived standards.  Like some “minimum recommended daily allowance” of a particular 21st century practice. This makes things much simpler for IMLS because the idea of library or museum standards, itself, is notoriously tricky.  Several of the approaches endorsed in their model don’t apply to many institutions.  (How can a small rural library or a historic police museum be collaborating with community partners on its new educational programs “over 75% of the time?”)

Fortunately, these types of measurement issues are immaterial.  Remember, this is not assessment.  It is education and proselytizing.  In fact, the IMLS self-assessment tool demonstrates one 21st century skill enhancement technique first-hand. As described in the project report, the tool is clearly interactive audience involvement! Rather than posting the questionnaire merely to measure something, IMLS is modeling the behavior they are seeking from museums and libraries.  I think it’s called “showing by doing.”

For one thing, sound cost-benefit analysis requires an examination of the full extent of relevant costs and benefits of a given project, program, or service. These quick-and-easy library calculators, however, use average retail prices as proxies for benefits. This oversimplification ignores important sources of library value like contributions to student and life-long learning, scientific and academic research, and public discourse, as well as roles libraries play in imparting cultural and humanitarian values and traditions, promoting literary appreciation and aesthetic values, facilitating community cohesion, and so forth.

But say that, for practical purposes, we accept the idea that value-boils-down-to-price as reasonable. Even so, the retail pricing approach these calculators use has definite problems. The calculators view retail prices as estimates of costs that patrons would incur if the library’s items and services were—hypothetically—unavailable to the community or institution. The library comes up with a retail price for each type of material and service it offers, and then these prices are translated directly into the value patrons receive from utilizing these materials or services.

In many cases, however, the alternative to obtaining an item or service from the library is not an outright purchase at retail prices. A student might purchase a textbook for $125 and then later re-sell it on Amazon.com for $50. Or perhaps she buys the item at a used price or borrows it from a friend for free. Clearly, a variety of alternative patron scenarios are possible, meaning that there is a range of alternative costs (approximate values) associated with each item or service use. The average of these ranges will typically be less than an item’s retail price. Besides, an item borrowed from a library does not include the breadth of rights and conveniences that item ownership does. So, it is a stretch to say that a patron always enjoys the same benefit from a borrowed item as from a purchased one.

Other problems with the calculators make their output suspect. For example, each time a patron renews an item or re-uses it in-house or online, the item’s retail price gets credited—again—to the library’s value totals. (Cha-ching!) On the other hand, when our Amazon.com shopper purchases a book at $75, that book’s value does not increase to $150, then to $225 and beyond each time the owner opens the book, or with each 3-week library loan period that passes.

Because the calculators tally only certain types of transactions, they end up painting a rather rosy picture of library performance. Consider the case of a patron who needs an item or service that is (really) not available from the library, and whose information need ultimately goes unmet. And the case of a service delivered that fails to meet a patron’s need, such as an unproductive reference consultation. The first case won’t be tallied at all by these calculators, and the second case will be tallied but will be significantly over-valued. (It will be considered a complete success.) Yet, the actual value of both of these patron transactions is negative and should be entered into these calculators this way. Unfortunately, the calculators’ designs do not accommodate this.

Given these problems and oversights, it is fairly obvious that these calculators produce exaggerated estimates of the benefits which libraries provide. Perhaps this exaggeration is only moderate or perhaps it is substantial—we cannot really tell for sure.

The calculators also underestimate the cost side of the equations, causing their benefit/cost ratios to be even more over-stated. They ignore several key costs incurred in delivering library materials and services, including expenses for information technology, equipment, building maintenance, utilities, and administrative overhead. These calculators also disregard the incidental costs that patrons may bear, like travel and parking costs, time lost due to item unavailability or poor service, usability difficulties encountered, and so on. In fact, NNLM’s calculator errors in the opposite direction: Assuming that libraries are always convenient, the calculator builds a patron time-savings factor into its formula. (I suppose you could enter in negative numbers to register patron lost time and inconvenience.)

When the calculators do recognize costs, they end up settling for data that are the grossest of estimates. For instance, users can enter estimated percent of total library staff time spent supporting access to materials or services. Creators of the calculators seem unaware that accurate benefit/cost ratios require meticulous collection of operational data, not just convenient guess-timates.

You will be hard-pressed to learn about these shortcomings from the materials that accompany library value calculators. Mostly, libraries receive general guidelines for entering data and encouragement to use the calculators without reservation. The library just keys in its data and—voilà!—receives an exact return-on-investment percentage or benefit/cost ratio right on the spot! Given the casual assumptions the calculations entail and the inexactness of the library’s input data, you’d think the final answer would at least include some type of margin-of-error disclaimer. Maybe something like this:

Your library’s benefit-to-cost ratio = $8.20 per $1.00 cost*

*    Based on our calculations, we are 95% confident that your library’s benefit/cost ratio is between $4.50 and $12.50 (per $1.00 cost). If your data are especially inaccurate, this range will be larger. Note that our single $8.20 estimate may be high due to assumptions our model uses.

Needless to say, this kind of small print doesn’t appear in the instructions that come with library value calculators. As they are, the calculators generate figures that are precise to the penny, with no other explanations to speak of. Libraries confidently report the figures to stakeholders as accurate, authoritative, and nearly approaching Scientific Truth. Of course, the figures are nothing of the sort.

Clones of these library calculators have sprouted up on dozens of library websites, where patrons are invited to enter their custom data to receive their own monthly “value of library services.” Costs are typically not mentioned, so that final value calculations are simple multiplications of counts times arbitrary and often fanciful retail price estimates. Of course, the nifty and optimistic totals will delight library patrons. The totals might even please the population of nonusers who are happy to subsidize library use by others as an overall benefit to the community or institution.

On a few public library websites the calculations are made even more tantalizing by informing patrons about their “individual return-on-investment”—how much value they gain for every tax dollar they contribute. (Don’t you just love democracy!) Unfortunately, this approach casts the wrong light on the public value of libraries. First, the figures are further exaggerations because they use per capita revenue data. Not every public library user pays taxes, a fact that makes the individually quoted return rates artificially high. (Instead of library tax revenue per capita, the calculations should use tax revenue per tax-paying household.)

Second, these seemingly benign “value” calculations actually hide information. The websites fail to provide overall return-on-investment rates for all tax-paying households or for all tuition-paying students. As I have already alluded, an individual patron’s rate of return is being subsidized by nonusers of library services. For every patron elated with his own personally-calculated rate, there will be several households or students whose return rates are less than $0, meaning they lose money on their library tax or tuition “investments.” (This mix of returns rates also applies to using the vanilla versions of the calculators that don’t bother to factor costs in.) Omitting this larger picture from these presentations is slanted and misleading—something that libraries should not be involved in.

From a public or institutional value perspective, these Library 2.0-inspired patron calculators completely sidestep the rightful purpose of library evaluation. This purpose is to assess the extent to which the library provides value to the institution or community as a whole, not how each individual fares. This assessment must also confirm that products and services are equitably distributed, that is, equally available and accessible to all who wish or need to use them (see Creating Public Value by Mark H. Moore).

In actuality, economic valuation is not so simple as it appears. It involves complicated (and frustrating) concepts like exchange value, use value, contingent value, and others. Even business corporations have misgivings about standard return-on-investment analysis because of how difficult it is to obtain reliable data to input into the formulas.

If we want to use purely monetary estimates of the value of our services, we need more rigorous methods than these makeshift library calculators. This exact advice was offered to us a couple of years ago by Donald Elliot and Glen and Leslie Holt in their book Measuring Your Library’s Value. Their work provides important guidance that we should be heeding. Like the fact that benefit/cost valuations are unique to the communities and institutions from whence they come. The figures are really not comparable across communities or for different libraries. This is something that most of us would not have thought about. The central message from their book, though, should already be obvious to us: We can’t just make these benefit/cost numbers up, the way these calculators do. There have to be sound theoretical and empirical bases for our findings.

Sure, quick-and-dirty estimations might be helpful in certain situations, as long as they are recognized for what they are. But the numbers gushing from these library calculators are nonsensical and disingenuous, in many cases. The whole idea has become an impediment to the real work of assessing library value. When the batteries in these little pocket library calculators wear out, I recommend that we just not replace them.

Instead, you might appreciate some other informational experience better, one that happens also to be thoroughly cool and engaging. Like Facebook walls or those omnitemporal slice-of-life Twitter tweets.

This post definitely is not slice-of-life. Hardly. It is conceptual, meaning that it is mostly tedious and definitely time-consuming.  It entails plodding through the text to see if any of the ideas make any sense. And even if they do, you have to figure out whether they are at all relevant. Worse, the topic could be one of those god-awfully amorphous ones that have no clear, calculatable bottom lines—like conundrums or Zen Buddhist koans.

Well, since you’re reading this paragraph, you must have free time on your hands.  So, I’ll tell you that the title of this post is from a National Public Radio essay by commentator Mark Allen.  Allen recounts how his boss insists that Allen send him only brief, concise email messages.  The boss apparently realizes that life is too short to get bogged down in details.  Or, God forbid, in the subtleties of precision, meaning, and context. Too much information. Shorter. Allen says that for people like his boss who subscribe to the Utopian vision of Life 2.0, “speed and brevity are obviously more important than facts, words, or information.”

Nowadays it is a social faux pas to communicate in long sentences with colleagues, friends, and family.  It’s self-indulgent, counter-productive, and so 20th century!  (Actually, I like to think of it as so 18th century since that’s when expository writing actually sprang up.)

Every so often, though, brevity and simple-minded factoids end up being extremely dangerous. I am thinking of the 2003 Columbia space shuttle accident that killed seven astronauts and crippled the NASA shuttle program.  The (I apologize) details about the role that sound-bite-like thinking played in this tragedy can be seen in the thoughtful work of data-presentation expert Edward Tufte.

Bottom line—the format in which information is presented has a gigantic effect on the information itself.  Marshall McLuhan’s famous quote ‘The medium is the message’ said essentially this.  Bottom lines filter out lots of information and it is never clear what crucial data have gotten omitted.  (Listen to the NPR audio to hear how the print version strips out information that is otherwise embedded in the single spoken word “shorter.”) In the case of textual information, simplified formats lead to simplified information.  Complicated ones enable the presentation of more complex and richer information.

Thankfully, the engineering details of space shuttle systems can be fairly well specified, as Tufte points out.  The task just requires ample formats for text, formulas, performance data, and diagrams that permit the exploration of the information, including its obvious and latent interrelationships.  And, of course, a commitment to studying and analyzing the information systematically.

Putting too many time and space restrictions on information distorts the information. But, as Allen notes, managers on a mission want bottom line answers.  They inhabit the world of perpetual motion and decisive action—not contemplation and analysis.  When a manager is seeking a tree, the forest can only be an aggravation.

Tufte has an almost scriptural response to the temptation to oversimplify important phenomena:  “It’s more complicated than that.” Which I will supplement with this verse: “Woe to the manager who under-contemplates a really important decision.”

All too often Tufte’s adage applies to informational practices in businesses and in public institutions, including libraries.  Short, over-simplified answers typically misrepresent the real situation. And they tend to justify the conduct of business as usual.  Responsible and effective public management (that is, stewardship of the public’s resources), however, requires a commitment to analyzing and digesting operational, performance, and environmental data, recognizing where informational gaps exist, identifying possible connections, looking for underlying logic, structure, and trends, and determining what relevant conclusions or generalizations can justifiably be drawn from these details. 

But all of this is a big hassle when there is more pressing work to be done.  Work like hunkering down to absorb library budget cuts, re-allocate staff, pare down materials costs, pay utility bills, deal with unions, and so on.  When we have more time, we’ll study our data to inform our decisions, and maybe even refine what we collect.  But right now we’re in a time crunch!