Lib(rary) Performance » Research

Stranger Than Fiction

raylyons — Wed, 17 Mar 2010 02:21:20 +0000

I proceed with this little essay with some trepidation due to the topic I’ve chosen: the idea of margins of error in survey research. By “survey research” I mean such things as political polls, public opinion surveys, market research, and so on.

Right up front I can share my conclusion with you. The common understanding of margins of error is incorrect. If you decide not to read this mini-exploration in its entirety, just remember this: Whatever you thought or heard that margins of error in surveys mean, simply abandon those ideas. Replace them with a mental question mark and you’ll be fine.

I say this because the precise meaning of margins of error is stranger than fiction. After all, the concept does come from statistical theory, which can definitely be on the fantastical side. Read on and you will see.

Navigating with Fragmentary Information

raylyons — Wed, 17 Feb 2010 00:23:04 +0000

I have implied this in other entries in this blog, but I might as well say it outright: The library and information science profession needs to come to terms with the issue of standards for (i.e., rules of) evidence for performance, statistical, and advocacy research data. There, now I’ve said it.

I recently read the short and enjoyable book Graphic Discovery: A Trout in the Milk and Other Visual Adventures by statistician Howard Wainer (Princeton, NJ: Princeton University Press, 2005). The subtitle of the book comes from something Henry David Thoreau wrote. During a dairy strike in 1850 in New England people began to suspect that dairy owners were watering down the milk supply. This led Thoreau to write in his journal, “Sometimes circumstantial evidence can be quite convincing; like when you find a trout in the milk” (quoted in Wainer, p. 81).

Wainer’s main point, one certainly made also by others like William Cleveland and Edward Tufte, is that well designed graphical representations are invaluable for exploring and understanding data. Graphical representation of data can lead to revelations about data and the underlying phenomena they describe that would otherwise be missed. [Read more...]

The Telephone Game

raylyons — Tue, 05 Jan 2010 13:34:47 +0000

Readers of the, say, older persuasion may recall a time when children actually enjoyed games that required no peripheral devices, infrared sensors, or satellite tracking. There was one party game, simply called (I think) “Telephone,” where one player whispered a message to the next, and that player to the next, until the message was passed all the way around the circle of players. The fun came when everyone heard the amusing distortions that ended up in the final message.

In library advocacy research, though, message distortion is not amusing. I noticed a serious instance of this in a recent IMLS Research Brief which cites an American Library Association (ALA) report finding that patron use of library computers for job-seeking purposes has “greatly increased.” [Read more...]

Research Literacy

raylyons — Fri, 11 Sep 2009 02:11:29 +0000

In 2008 the Online Computer Library Center (OCLC) published a marketing research report addressing the need for increasing public support for libraries. The study, From Awareness to Funding: A Study of Library Support in America, was funded by a $1.2 million grant awarded by the Bill & Melinda Gates Foundation.

Respondents in the study were divided into four groups according to how supportive of public library funding they were. Somehow—the OCLC researchers don’t say how—certain questionnaire responses qualified respondents for assignment to each group. But the results don’t always make sense. For instance, 20% of “super-supporter” group said they were not definitely committed to voting in favor of libraries. And 6% of this group was either unsure how they’d vote or said they would vote “No.” Wondering how such uncommitted respondents ended up assigned to the group that is “super” supportive of libraries, I contacted the OCLC researchers. Unfortunately, I never did get a response to this question. [Read more...]

Thoroughly Modern Museums and Libraries

raylyons — Mon, 31 Aug 2009 22:18:54 +0000

I think I get it now. I had thought the term assessment meant a systematic and appropriately rigorous measurement of a construct or phenomenon of interest, like program outcomes, community needs, service quality, and so on. Only now have I come to understand that a self-assessment is a different animal altogether. Who would have thought that the purpose of a self-assessment is not really to assess anything? The purpose, I now realize, is to inform and educate. All this time I have been applying research methodology standards to tools that are intended to advocate and indoctrinate. No wonder my observations have been so off-base!

When I critiqued WebJunction’s online competencies assessment questionnaire (see my April 22, 2009 entry), the WebJunction staff explained to me that the true objective for their surveys was to increase awareness of these competencies. I immediately wondered, “Well, how then will WebJunction measure awareness?” But that is quite an irrelevant question when these ques- tionnaires are actually teaching tools, not measurement instruments. [Read more...]

Cha-Ching!

raylyons — Fri, 14 Aug 2009 16:34:22 +0000

I noticed that yet another library value calculator has appeared on the scene. This one is offered by the National Network of Libraries of Medicine (NNLM) with the very best of intentions, I am sure. But, let me say that I am convinced that these calculators are a bad idea. Their underlying assumptions are weak and their designs are not well thought out. Eventually, library funders and stakeholders are going to realize that the calculations are superficial and…well…sloppy.

For one thing, sound cost-benefit analysis requires an examination of the full extent of relevant costs and benefits of a given project, program, or service. These quick-and-easy library calculators, however, use average retail prices as proxies for benefits. This oversimplification ignores important sources of library value like… [Read more...]

A Preponderance of the Evidence

raylyons — Tue, 30 Jun 2009 02:07:33 +0000

It is fairly well known that the field of business management can be susceptible to fads. Organizational scientists have studied the adoption of business approaches like management-by-objectives, total quality management (TQM), business process re-engineering, just-in-time manufacturing, scorecard methods, and others. Their work has led to an interesting body of literature about management innovations and organizational change.

One idea from this literature is that management innovations can morph from the original ideas of their founders. Over time TQM began to promote practices that quality gurus like W. Edwards Deming warned against, for instance, bestowing individual rewards for quality objectives accomplished. And sometimes organizations take liberties with the specifics of an innovation. They might
Dr. Deming decide to use only the components they’re most comfortable with or add their own idiosyncratic twists. [Read more...]

Once Size Doesn’t Fit All

raylyons — Thu, 07 May 2009 03:59:45 +0000

A basic tenet of public librarianship is the idea that each library and its communities are unique. While libraries share certain characteristics in common, their products, services, and operations are (in theory) highly customized to fit local conditions. I didn’t realize how strong a tenet this was until I heard this declaration at an Ohio Library Council conference: “All library excellence is local.” Wow, pretty unequivocal! Granted, public libraries do acknowledge that they have certain things in common with other libraries, but it sure sounds like unique characteristics trump everything else.

This contrast between things standard and things tailored (or customized) turns out to be a theme central to evaluation research also. The idea has been noted, for instance, by Mark Lipsey, co-author of the leading textbook on program evaluation. [Read more...]

Objects in Mirror Are Closer Than They Appear

raylyons — Fri, 01 May 2009 15:06:12 +0000

In January my brother and I were laying laminate flooring in his house. Each time we needed to trim a plank, we stood reverently by his table saw and incanted the familiar carpenter’s adage, “Measure twice, cut once. (Amen.)” My brother said, “It’s the damnedest thing. You can repeat and repeat a measurement, and then find out it is still wrong.” As an electrical engineer (he’s working on the 3rd edition of his book on digital signal processing), his observation comes from dozens of real-life technical projects.

In the behavioral sciences as well as in program evaluation and performance assessment we attempt to measure fairly abstract things—like social class, anxiety, customer loyalty, community need, awareness of services, and so on. Measuring these is difficult. But even in the “hard” sciences measurement is a continuous challenge. [Read more...]

New (or Old?) Paradigm Spurs ‘Fundamental Shift’ in Library Advocacy

raylyons — Sat, 25 Apr 2009 07:23:37 +0000

“Everything Old is New Again” is the title of a 1999 article in American Libraries by Douglas Raber, author of the excellent and eye-opening book, Librarianship and Legitimacy: The Ideology of the Public Library Inquiry. The article sug- gests that the Inquiry, a comprehensive assessment of public librarianship initiated by ALA in the late 1940′s, continues to be relevant to libraries today. While in library school I discovered Raber’s book in the stacks of Cleveland Public Library. The book was so inspiring that I got ahold of 3 of the 7 volumes of the Inquiry (thank you, CPL!) and read them also.

Now the next piece in my story: My colleague Keith Curry Lance had recom- mended a podcast series to me. It is called “Longshots” and is broadcast by Sarah Long, Executive Director of the North Suburban Library System outside Chicago. I decided to take a listen and chose a December 2008 interview with Cathy de Rosa and Jenny Johnson, primary authors of the OCLC study, From Funding To Awareness: A Study of Library Support in America. A couple of months ago I studied the first half of this voluminous and highly graphicized report. In case I never got back to the second half, I thought I’d see how the audio book version went. [Read more...]