Lib(rary) Performance » Library assessment

Indentured Certitude

raylyons — Wed, 22 Feb 2012 19:20:50 +0000

I want to share some information with you from a resource I mentioned last month. The resource is Edward Suchman’s 1967 book, Evaluative Research and the information is this diagram, which presents a basic model of evaluation:1

I share the diagram because it presents two ideas that don’t always percolate to the top of discussions of library outcome assessment. The first idea is the need for programmatic values to be made explicit beforehand. Suchman, who worked in the public health field, gave this example:

Suppose we begin with the value that it is better for people to have their own teeth rather than false teeth. We may then set our goal that people shall retain their teeth as long as possible.2

Of course, it’s quite possible to hold different values. For instance, one might prefer false teeth over natural ones… [Read more]

—————————

1 Suchman, E. A. (1967). Evaluative research: Principles and practice in public service and social action programs, New York: Russell Sage, p.34.

Smaller is Beautifuler

raylyons — Sat, 18 Jun 2011 02:24:52 +0000

A recent article in AL Direct entitled The Smartest Readers presents some simple library rankings based on that stalwart library measure, circulation per capita. Rankings like these are, at least to me, a reminder of a perennial conundrum concerning the meaning of per capita library measures. For more than a century librarianship has puzzled over how to evaluate these statistics. Do per capita data tell us whether or not libraries are doing a good job? What amounts of materials made available or levels of services delivered are sufficient for libraries with specific missions and serving communities of a particular size and makeup?

Mainly, libraries have to rely on their own ingenuity to interpret per capita or per constituent data (like per student, faculty, employee, subscriber, stakeholder, and such). About the only official guidance they have gotten over the decades is advice about comparing (benchmarking) their data with appropriate peer libraries. Lacking some more objective gauge of statistical performance, libraries end up applying what might be called the more-is-better rule. [Read more...]

Discussing Accuracy

raylyons — Tue, 11 Jan 2011 22:07:29 +0000

Now that I am no longer distracted by the subject of last week’s entry I can get to the intended topic for my first 2011 blog entry. I should say, though, that I won’t be turning over any kind of new leaf for the new year. For now I’m sticking with the theme I’ve dwelt on already, a theme statistician Howard Wainer expressed concisely:

Whenever we discuss information we must also discuss its accuracy.1

Applied to the library world the message is: Formal library research and advocacy studies should always explain the strengths and limitations of their data. The best reason for abiding by this principle, in my opinion, is to allow readers to decide how much credence they want to give to conclusions drawn in the studies.

With new library advocacy studies on the horizon, I thought I better wrap up any unfinished topics from 2010. Howard Wainer’s advice made me think of the study, Opportunity for All: How the American Public Benefits from Internet Access at U.S. Libraries. Findings from this report have been broadcast widely with barely a mention of how approximate the figures are… [Read more]

—————————

1 Wainer, H., 2009, Picturing the Uncertain World: How to Understand, Communicate, and Control Uncertainty through Graphical Display, Princeton, NJ: Princeton University Press, p. 121.

Blog Health-o-Meter™

raylyons — Wed, 05 Jan 2011 01:10:43 +0000

I had a different topic in mind for this, my first post of 2011. But an email I received from the friendly staff at WordPress changed my mind. What good news! WordPress congratulated me on my blog’s attainment of a rating of Awesome on their Blog Health-O-Meter™. And I know they really approve of the job I am doing because they added, “We think you did great!” Of course, they assured me that their rating is completely data-based.

This surprising honor is cause for celebration, indeed! And I am gratified to be able to share it with you, my readership! I could never have done it without you! I am humbled and otherwise speechless… [Read More]

A Hard Row to Hoe

raylyons — Fri, 05 Nov 2010 20:39:50 +0000

During the question-and-answer part of a presentation I gave at the 2010 Library Assessment Conference in Baltimore last week, I couldn’t resist editorializing about how bad convenience sampling is. One audience member spoke up, saying she felt convenience samples are legitimate as long as findings are interpreted as describing only the respondents, themselves. Later on I realized she was making a more interesting point, something that . . . [Read more]

Navigating with Fragmentary Information

raylyons — Wed, 17 Feb 2010 00:23:04 +0000

I have implied this in other entries in this blog, but I might as well say it outright: The library and information science profession needs to come to terms with the issue of standards for (i.e., rules of) evidence for performance, statistical, and advocacy research data. There, now I’ve said it.

I recently read the short and enjoyable book Graphic Discovery: A Trout in the Milk and Other Visual Adventures by statistician Howard Wainer (Princeton, NJ: Princeton University Press, 2005). The subtitle of the book comes from something Henry David Thoreau wrote. During a dairy strike in 1850 in New England people began to suspect that dairy owners were watering down the milk supply. This led Thoreau to write in his journal, “Sometimes circumstantial evidence can be quite convincing; like when you find a trout in the milk.” [Read more...]

Sawing with a Dull Saw

raylyons — Mon, 25 Jan 2010 18:35:42 +0000

In spite of their evolution over the last few decades, accelerated most recently due to the Googlization of information, public libraries have been amazingly impervious to change in the arena of performance measurement. I found the following observations about library measures in the early history of American libraries:

There is no branch of library economy more important, or so little understood by a librarian as helps to himself, as the daily statistics which he can preserve of the growth, loss, and use (both in extent and character) of the collection under his care. The librarian who watches these things closely, and records them, always understands what he is about, and what he accomplishes or fails to accomplish. The patrons to whom he present these statistics will comprehend better the machinery of the library, and be more indulgent toward its defects.1

Interesting that use of library statistics for advocacy purposes was recognized in 1876. [Read more...]

—————————

1 Warren, S. R. & Clark, S. N., Eds., 1876, Public Libraries in the United States of America, Washington, DC: U.S. Bureau of Education, p. 714.

Thoroughly Modern Museums and Libraries

raylyons — Mon, 31 Aug 2009 22:18:54 +0000

I think I get it now. I had thought the term assessment meant a systematic and appropriately rigorous measurement of a construct or phenomenon of interest, like program outcomes, community needs, service quality, and so on. Only now have I come to understand that a self-assessment is a different animal altogether. Who would have thought that the purpose of a self-assessment is not really to assess anything? The purpose, I now realize, is to inform and educate. All this time I have been applying research methodology standards to tools that are intended to advocate and indoctrinate. No wonder my observations have been so off-base! [Read more...]

Cha-Ching!

raylyons — Fri, 14 Aug 2009 16:34:22 +0000

I noticed that yet another library value calculator has appeared on the scene. This one is offered by the National Network of Libraries of Medicine (NNLM) with the very best of intentions, I am sure. But, let me say that I am convinced that these calculators are a bad idea. Their underlying assumptions are weak and their designs are not well thought out. Eventually, library funders and stakeholders are going to realize that the calculations are superficial and…well…sloppy.

For one thing, sound cost-benefit analysis requires an examination of the full extent of relevant costs and benefits of a given project, program, or service. These quick-and-easy library calculators, however, use average retail prices as proxies for benefits. This oversimplification ignores important sources of library value like… [Read more...]

Shorter

raylyons — Mon, 20 Jul 2009 05:56:24 +0000

You may not want to spend time reading this blog post. It’s rather long and drawn out and is likely to be dull. And it gets kind of complicated. Besides, the graphics are sparse and uninteresting. Plus there’s no video.

Instead, you might appreciate some other informational exper- ience better, one that happens also to be thoroughly cool and engaging. Like Facebook walls or those omnitemporal slice-of-life Twitter tweets.

This post definitely is not slice-of-life. Hardly. It is conceptual, meaning that it is mostly tedious and definitely time-consuming. It entails plodding through the text to see if any of the ideas make any sense. And even if they do, you have to figure out whether they are at all relevant…. [Read more...]