Lib(rary) Performance Blog

Entries from January 2010

Sawing with a Dull Saw

January 25, 2010 · Leave a Comment

In spite of their evolution over the last few decades, accelerated most recently due to the Googlization of information, public libraries have been amazingly impervious to change in the arena of performance measurement. I found the following observations about  library measures in the early history of American libraries:

There is no branch of library economy more important, or so little understood by a librarian as helps to himself, as the daily statistics which he can preserve of the growth, loss, and use (both in extent and character) of the collection under his care. The librarian who watches these things closely, and records them, always understands what he is about, and what he accomplishes or fails to accomplish. The patrons to whom he present these statistics will comprehend better the machinery of the library, and be more indulgent toward its defects.     Public Libraries in the United States of America, Warren, S.R. And Clark, S. N., Eds., Washington, DC: U.S. Bureau of Education, 1876, p. 714.

Interesting that use of library statistics for advocacy purposes was recognized in 1876.

Early in the twentieth century our current ideas about performance measurement were already well understood, long before the practices of program evaluation, evidence-based management, and performance scorecarding were formalized. Arthur Bostwick, late director of St. Louis Public Library and Librarian of Brooklyn Public Library, wrote this in his book published in 1917:

No business can be properly carried on without a system of accounts. These may involve only money received and expended, but they may and should extend much further. The collection and tabulation of such [financial and performance] data have come to be regarded as indispensable by shrewd businessmen; and large corporations do not hesitate to spend considerable sums in employing a force of experts and clerks especially to gather data of this kind and to tell what they mean…    The American Public Library, Bostwick, A.E., New York: D. Appleton & Co., 1917, p. 253.

Bostwick also dealt with the ideas of accountability, continuous improvement, careful analysis of library statistics, and a tiered approach to evaluation data-collection:

Information of this [financial and performance] kind is gathered with either or both of two different purposes in view—to satisfy the legitimate curiosity of the person managing the business, or of some one who has a right to know how it is going on, whether it is succeeding or failing and just what it is accomplishing; and, secondly, to furnish a basis for improvements or changes, to indicate weak points and points of strength, so that the business may be reënforced along the former and extended along the latter.

…If the latter [purpose is intended], a more detailed and analytical study is made of the data, which are compared and tested in all possible ways to reveal unsuspected facts. When something is thus brought to light that seems to call for further investigation, additional data are collected. (p. 254)

He also preached about the imprecision of library statistics, a topic conveniently overlooked in our profession nowadays (as is the importance of assuring the validity of data presented in library advocacy reports also):

It should not be forgotten, either by those who collect and report these statistics, or by those who read them or use them, that they are of various degrees of exactness…In any kind of scientific measurement the limits of probable error are always mentioned to give an idea of the degree of accuracy. The less the probable error, the greater the accuracy. It is never stated that there can be no error and that the accuracy is exact; this would be simply ridiculous. The same holds good in library statistics. In the average report nothing at all is said of accuracy; the reader is left to conclude that all the data are exact, or at least that there is no difference in their degree of exactness. (p. 262)

Finally, the level of interest in this topic among Bostwick’s 20th century peers strikes a familiar chord:

But how much intelligent study of library statistics goes on in librarian’s offices, and how much modification or improvement in library methods and material results from such study, is something that we shall never now. It appears to be certain, however, that large numbers of librarians…look upon their statistics in the light of a necessary evil. They must be collected, because some thing of the kind is expected in the annual report, but they should be minimized, and, once in print, they should be dismissed from the mind. This attitude reminds one of the rural workman who used a dull saw because the amount of work before him gave him no time to stop and sharpen it… (p. 255)

Categories: Library assessment · Measurement
Tagged: , ,

The Telephone Game

January 5, 2010 · Leave a Comment

Readers of the, say, older persuasion may recall a time when children actually enjoyed games that required no peripheral devices, infrared sensors, or satellite tracking. There was one party game, simply called (I think) “Telephone,” where one player whispered a message to the next, and that player to the next, until the message was passed all the way around the circle of players. The fun came when everyone heard the amusing distortions that ended up in the final message.

In library advocacy research, though, message distortion is not amusing. I noticed a serious instance of this in a recent IMLS Research Brief which cites an American Library Association (ALA) report finding that patron use of library computers for job-seeking purposes has “greatly increased.” The ALA report is Job-Seeking in U.S. Public Libraries and the statement the IMLS brief cites is this:

“As part of site visits to public libraries in nine states [conducted in three annual studies], the study research team has found greatly increased use of library technology for job-seeking and e-govern-ment.” Job-Seeking in U.S. Public Libraries, American Library Association, Oct. 2009, p. 2.
(The emphasis in red is mine).

The ALA report is one of a series of “issue briefs” published by the Office for Research and Statistics that summarize and supplement key results from the multi-year Public Library Funding and Technology Access study. To date this project has issued three annual reports beginning with the 2006/2007 edition. (The project is a collaborative effort connected with the Public Libraries and the Internet longitudinal studies which began in 1994.)

Anyway, I wondered how big this increase actually was and what the level of job-seeking computer use had been before the big increase happened. So I went searching for the numbers in the ALA Public Library Funding annual studies. Turns out none of the studies measured frequency of job-seeking or e-government computer use by patrons. Nor did the studies compare frequencies of any reported computer uses from year to year. The 2006/2007 and 2007/2008 editions merely state that job-seeking and e-government were common uses reported by some patrons, without mentioning increases of any sort. The 2008/2009 edition reports increased patron job-seeking computer use, but does not describe this increase as substantial or “great.”

The quotation from the issue brief (above) says that the researchers detected this increase by means of interviews with staff and patrons during library site visits. These interviews, conducted in a few selected U.S. states each year, included a simple open-ended question to patrons: “What do you use [the library's computers] for?” Job-seeking and e-government made the lists of most frequent responses (and each state’s list apparently differed from the others). But, no frequency counts for these uses show up in the three studies, perhaps because the counts weren’t collected. (Site visits, as well as focus groups which the ALA studies held, belong to the category of qualitative research methods. As the ALA project illustrates, collecting essentially quantitative information using qualitative methods can lead to problems.)

Even if the researchers did tally these uses during the interviews, neither the interviewees nor the states where interviews were conducted were randomly selected. So, we couldn’t say the tallies represent the larger patron population nationally. The 2007/2008 study reports a convenience sample of about 200 patrons who were using library computers at the time (Libraries Connect Communities, American Library Association, 2008, p. 128). (See the NOTE below.) The same study then reports that “Interviews with users confirmed staff observations that much computer use in libraries is job-related…” (p. 131; I added the emphasis). Exactly how much they don’t say. And how often unrepresented patrons—say teenagers who had yet to show up after school—might use computers for job-seeking we can’t tell. (The 2006/2007 ALA study cites a 2006 Baltimore, MD study where library computer use was found to depend on age group. See footnote on p. 169 of the ALA 2006/2007 study.)

Selection bias of a different sort confounds year to year comparisons from the ALA studies. Because the annual interviews were conducted in different states where the job markets and online government services could differ significantly, there are no reliable baselines for comparing job-seeking and e-government computer usage between years. For instance, the 2006/2007 study included a site visit to a library in Nevada where staff reported long lines of patrons using library computers to apply for jobs at a newly opened gambling casino. Relying on such atypically high usage as a baseline for job-seeking computer use could mask actual increases in later study years in different states.

The question remains, what data are these “great increases” based on? None that I can find in the ALA studies. The issue brief does cite other figures that did increase over time, but these figures don’t describe patterns of computer use. The figures are from the survey portion of ALA’s studies, from questionnaire items that elicit library staff opinions. Staff were asked to identify the top five public Internet services that they believed to be “the most critical to the role of the library branch in its local community.” In the 2006/2007 study 44% of responding staff chose provision of job-seeking services as one of their top five priorities. In the 2007/2008 study this proportion was 62.2%, and in 2008/2009 it was 65.9%.

Basically, about 22% more votes by staff (each staff respondent got up to 5 votes) went to patron job-seeking services in the 2008/2009 survey than in the 2006/2007 one. While these vote tallies may reflect changes in staff perceptions of computer use during this period, the votes are only opinions, and don’t indicate how patrons actually used computers in libraries.

The truth is that we don’t have the usage data needed to support the assertion made in the ALA issue brief. Without valid baseline data, we can’t measure increases in patron job-seeking or e-government computer use at all, and we certainly can’t tell whether or not any increases have been great.

———————————-

NOTE:    Convenience sampling is a type of ‘nonprobability sampling.’ With nonprobability sampling, we have no statistical basis for claiming that our study findings describe the larger population that we had hoped our research would apply to. Using nonprobality sampling invites biased information into study results.

Categories: Advocacy · Measurement · Research
Tagged: , , , ,