Lib(rary) Performance » Library assessment

Navigating with Fragmentary Information

raylyons — Wed, 17 Feb 2010 00:23:04 +0000

I have implied this in other entries in this blog, but I might as well say it outright: The library and information science profession needs to come to terms with the issue of standards for (i.e., rules of) evidence for performance, statistical, and advocacy research data. There, now I’ve said it.

I recently read the short and enjoyable book Graphic Discovery: A Trout in the Milk and Other Visual Adventures by statistician Howard Wainer (Princeton, NJ: Princeton University Press, 2005). The subtitle of the book comes from something Henry David Thoreau wrote. During a dairy strike in 1850 in New England people began to suspect that dairy owners were watering down the milk supply. This led Thoreau to write in his journal, “Sometimes circumstantial evidence can be quite convincing; like when you find a trout in the milk” (quoted in Wainer, p. 81).

Wainer’s main point, one certainly made also by others like William Cleveland and Edward Tufte, is that well designed graphical representations are invaluable for exploring and understanding data. Graphical representation of data can lead to revelations about data and the underlying phenomena they describe that would otherwise be missed. [Read more...]

Sawing with a Dull Saw

raylyons — Mon, 25 Jan 2010 18:35:42 +0000

In spite of their evolution over the last few decades, accelerated most recently due to the Googlization of information, public libraries have been amazingly impervious to change in the arena of performance measurement. I found the following observations about library measures in the early history of American libraries:

There is no branch of library economy more important, or so little understood by a librarian as helps to himself, as the daily statistics which he can preserve of the growth, loss, and use (both in extent and character) of the collection under his care. The librarian who watches these things closely, and records them, always understands what he is about, and what he accomplishes or fails to accomplish. The patrons to whom he present these statistics will comprehend better the machinery of the library, and be more indulgent toward its defects. Public Libraries in the United States of America, Warren, S.R. And Clark, S. N., Eds., Washington, DC: U.S. Bureau of Education, 1876, p. 714.

Interesting that use of library statistics for advocacy purposes was recognized in 1876. [Read more...]

Thoroughly Modern Museums and Libraries

raylyons — Mon, 31 Aug 2009 22:18:54 +0000

I think I get it now. I had thought the term assessment meant a systematic and appropriately rigorous measurement of a construct or phenomenon of interest, like program outcomes, community needs, service quality, and so on. Only now have I come to understand that a self-assessment is a different animal altogether. Who would have thought that the purpose of a self-assessment is not really to assess anything? The purpose, I now realize, is to inform and educate. All this time I have been applying research methodology standards to tools that are intended to advocate and indoctrinate. No wonder my observations have been so off-base!

When I critiqued WebJunction’s online competencies assessment questionnaire (see my April 22, 2009 entry), the WebJunction staff explained to me that the true objective for their surveys was to increase awareness of these competencies. I immediately wondered, “Well, how then will WebJunction measure awareness?” But that is quite an irrelevant question when these ques- tionnaires are actually teaching tools, not measurement instruments. [Read more...]

Cha-Ching!

raylyons — Fri, 14 Aug 2009 16:34:22 +0000

I noticed that yet another library value calculator has appeared on the scene. This one is offered by the National Network of Libraries of Medicine (NNLM) with the very best of intentions, I am sure. But, let me say that I am convinced that these calculators are a bad idea. Their underlying assumptions are weak and their designs are not well thought out. Eventually, library funders and stakeholders are going to realize that the calculations are superficial and…well…sloppy.

For one thing, sound cost-benefit analysis requires an examination of the full extent of relevant costs and benefits of a given project, program, or service. These quick-and-easy library calculators, however, use average retail prices as proxies for benefits. This oversimplification ignores important sources of library value like… [Read more...]

Shorter

raylyons — Mon, 20 Jul 2009 05:56:24 +0000

You may not want to spend time reading this blog post. It’s rather long and drawn out and is likely to be dull. And it gets kind of complicated. Besides, the graphics are sparse and uninteresting. Plus there’s no video.

Instead, you might appreciate some other informational exper- ience better, one that happens also to be thoroughly cool and engaging. Like Facebook walls or those omnitemporal slice-of-life Twitter tweets.

This post definitely is not slice-of-life. Hardly. It is conceptual, meaning that it is mostly tedious and definitely time-consuming. It entails plod- ding through the text to see if any of the ideas make any sense. And even if they do, you have to figure out whether they are at all relevant. Worse, the topic could be one of those god-awfully amorphous ones that have no clear, calculable bottom lines—like conundrums or Zen Buddhist koans. [Read more...]

Library Assessment 101

raylyons — Tue, 19 May 2009 00:11:46 +0000

I want to communicate what I believe is the single most useful message about library assessment. This is not an announcement of a new data analytic technique or some all-purpose library value calculator. Nor is it advice on the importance of aligning work and measurement with vision and strategy, recognizing the political pitfalls of evaluation, or solidifying an annual assessment plan.

All of these are secondary to one fundamental step. But this step is a giant one: Libraries must become “self-evaluating organizations.” The importance of this dawned on me (again) when I heard a librarian describing how her library used customer surveys to rethink their service approach. I realized it was not their survey questionnaire nor the planned service changes that mattered. It was their whole mindset that made the difference. They had a willingness to be inquisitive and exploratory, to be logical and systematic, to question comfortable assumptions, to look for unexpected answers, and to act on what they learned. [Read more...]

Once Size Doesn’t Fit All

raylyons — Thu, 07 May 2009 03:59:45 +0000

A basic tenet of public librarianship is the idea that each library and its communities are unique. While libraries share certain characteristics in common, their products, services, and operations are (in theory) highly customized to fit local conditions. I didn’t realize how strong a tenet this was until I heard this declaration at an Ohio Library Council conference: “All library excellence is local.” Wow, pretty unequivocal! Granted, public libraries do acknowledge that they have certain things in common with other libraries, but it sure sounds like unique characteristics trump everything else.

This contrast between things standard and things tailored (or customized) turns out to be a theme central to evaluation research also. The idea has been noted, for instance, by Mark Lipsey, co-author of the leading textbook on program evaluation. [Read more...]