Lib(rary) Performance

Paved with Good Intentions

raylyons — Tue, 09 Apr 2013 12:47:47 +0000

It never hurts to revisit the basics of a method that we’ve chosen to apply to a task we want to accomplish or a problem needing solved. So, the recent announcement of the Library Edge benchmarks is a good occasion to discuss that particular performance assessment method. In the third edition of his book, Municipal Benchmarks, University of North Carolina professor David Ammons describes three types of benchmarking:1

1. Comparison of performance statistics
2. Visioning initiatives
3. “Best practices” benchmarking

The idea behind item #1 is that the sufficiency of an organization’s performance can be judged by comparing its performance data with other organizations or against externally defined standards. Comparisons of different organizations using only performance data, without any reference to standards, is called comparative performance measurement. An Urban Institute handbook of the same name by Elaine Morley, Scott Bryant, and Harry Hatry gives an in depth explanation of this method.2 [Read more...]

—————————

1 Ammons, D. N. (2012). Municipal benchmarks: Assessing local performance and establishing community standards, Armonk, NY: M.E. Sharpe, p. 15.
2 Morley, E., Bryant, S. P., & Hatry, H. P. (2001). Comparative performance measurement, Washington, DC: The Urban Institute Press.

Bad Arrangements To Place Before School Children

raylyons — Sat, 09 Feb 2013 22:57:42 +0000

To begin this episode I want to introduce you to a couple of historical ideas on best practices in graphical data presentation—or using the more modern term, data visualization. (The peculiar title I’ve chosen comes from this history. Read on to see what it means.) Then I’ll step through a redesign of a bar chart to show you how effective graphical simplicity can be.

In the 1980’s and 1990’s statistical graphing experts Edward Tufte, William Cleveland, and Howard Wainer were promoting fair and clear designs for statistical charts.1 Nearly seventy years earlier American engineer Willard C. Brinton was doing the same thing in his 1914 book, Graphic Methods for Presenting Facts. Here’s a figure from the book:

Source: Brinton (1914), Graphic Methods for Presenting Facts, p. 21

Note that in his figure Brinton advocated for “accuracy of statement.” He did the same in this next… [Read more]

—————————
1 A more recent and quite definitive book on the principles of best data visualization practice is the second edition of Stephen Few’s book, Show Me the Numbers.

Data Detour

raylyons — Fri, 18 Jan 2013 18:03:49 +0000

Nowadays libraries aspire to be data-driven. Almost everyone agrees that collecting and using data to improve organizational performance is a good thing. Implied in the various regimens promoting this idea (library assessment, managing-for-results, evidence-based practice, quality management, etc.) is the need for practicing two virtues: patience and determination. These virtues happen also to be part and parcel of information literacy. We repeatedly advise users to avoid settling for the most convenient and quickly accessible information that shows up. And urge them to put ample time and energy into thinking critically about their question, its context, and the complete range of potentially relevant information.

Being data-driven requires this same discipline. I mention discipline because this blog entry concerns a challenging but quite educational quantitative topic. If you follow this to its conclusion I guarantee your numeracy muscles will be invigoratingly exercised!

I begin with this caveat: When advocacy or public relations professionals fail to think critically about their data, they risk communicating the wrong message to their audiences. Last month I ran across an interesting case of this in a radio public service announcement by the Ohio Highway Patrol . . . [Read more]

Oh The Weather Outside Is Frightful!

raylyons — Wed, 05 Dec 2012 22:41:42 +0000

The graphic below is a variant of one I blogged about in my prior entry. Its designers added a storm to create what might be called an inclement tug-of-war. This version of the graphic is from the Libraries Connect Communities: Public Library Funding & Technology Access Study 2011-2012 (PLFTAS) press kit. Both versions communicate this same basic message: The Great Recession led to severe and cumulative cuts in library funding and, at the same time, to an unrelenting surge in demand for traditional and technological services. The supporting data are the same in both versions. As I explained last time, since the data are not measures of actual library funding or usage, they don’t really confirm the graphic’s claims.

Source: ALA, Libraries Connect Communities, 2012. Click for larger image.

Actually, there are funding and usage data that can shed light on these claims, namely the Institute for Museum and Library Services (IMLS) survey data collected annually from 9000+ U.S. public libraries.1 I decided to examine data for 2005 through 2010 from this dataset.2 These cover the official span of the Great Recession (December 2007 to June 2009) with some leeway to identify trends that were ongoing prior to the recession.

First, I’ll summarize what the data show and then follow with details. For U.S. public libraries as a whole, funding cuts during the Great Recession were not nearly the calamity that library advocacy stories described, nor were the cuts cumulative over multiple years. Overall funding grew moderately through the recession and then began falling in 2010 . . . [Read more]

—————————

1 Each survey year has different counts of libraries reporting specific measures.
2 The 2010 public library survey data, issued last June, is the most recent dataset available from IMLS.

Statistical Hearsay

raylyons — Sun, 14 Oct 2012 02:50:55 +0000

I admit it. I’ve been suffering from a case of statistician’s block. No inspiring ideas for this blog have presented themselves since July. Well, actually, a couple did surface but I resisted them. Very recently, though, the irresistible “infographic” shown here came to my attention. I am therefore pleased to return to my keyboard to discuss this captivating image with you!

Source: ALA, Libraries Connect Communities, 2012. Click for larger image.

The infographic appears in the executive summary of the American Library Association’s (ALA) report, Libraries Connect Communities: Public Library Funding & Technology Access Study 2011-2012, published in June. The graphic’s basic message is an ongoing struggle between two sides. On the left the blue silhouetted figures represent public demand for technology services at libraries, with four percentages quantifying levels of use. The lone silhouette on the right side personifies library funding (is he a municipal budget official?), with a single percentage quantifying that. Apparently, the quantities on the left are, using the tug-of-war metaphor, overpowering the right side.

Let’s look a bit closer at the quantitative evidence in this infographic . . . [Read more]

Honest-to-Goodness Transformation

raylyons — Fri, 06 Jul 2012 17:30:54 +0000

A while back, in his 21st Century Library Blog Steve Matthews commented on some data appearing in a report entitled The Library in the City published by the PEW Charitable Trusts Philadelphia Research Initiative. Dr. Matthews was puzzled by an inconsistency between statistical trends highlighted in the report and standard per capita circulation, visits, and Internet computer measures. He noted, for example, that among the libraries studied Columbus Metropolitan Library had the greatest cumulative decline in visits (-17%) over the seven year study period. Yet, in 2011 Columbus ranked 2nd in the group on visits per capita. The opposite was true for the Enoch Pratt Library in Baltimore. Although the library showed the second highest cumulative increase in visits (at 25%), its 2011 per capita visit rate was the lowest in the group. Curious patterns, indeed.

There are a couple of statistical dynamics at play here . . . [Read more]

Assessment’s Top Models

raylyons — Mon, 21 May 2012 17:18:12 +0000

I recently attended a library webinar where the question of the difference between outputs and outcomes came up. The main idea was that outputs are programs and services an organization delivers, whereas outcomes are changes that occur in recipients, or their life situations, as a result of having received program services. Another was that outputs are distinguished by their more specific focus compared with outcomes, which are more general in scope. When I heard this second idea, it seemed correct in a way but incorrect in another. Mulling this over later, I began to wonder whether the first idea is not quite right, either.

To explain these new definitional doubts I’m having, I’ll need to review a couple of evaluation models with you. But first I’d like to clear something up. Just because some expert somewhere has drawn a diagram with rectangles and arrows and concise labels and called it a “model” doesn’t mean her/his creation is true, or even remotely so. Models are only true if . . . [Read more]

Fun With Numbers

raylyons — Sun, 01 Apr 2012 23:16:01 +0000

After so much stuff about evaluation theory and practice in this blog, it’s time for some fun! And what better fun is there than fun with numbers?1

Let’s begin our diversion with a graph from my prior post shown here. Looking closely, notice how some of the gold circles lie in neat, parallel bands. These bands

Click for larger image. Rest cursor over any circle in larger image to see individual library data. Data Source: IMLS 2009 Public Libraries Datafiles.

are more obvious in next two charts, which ‘zoom in’ on the data by decreasing the vertical axes value ranges. When I first saw this pattern, I suspected that something had corrupted the data. Double-checking, I found the data were fine, or at least they were true to the values in the original IMLS datafile. So, I decided to resort to that popular and trusty problem-solving technique . . . [Read more]

—————————
1 No, this is not an April Fool’s joke. I propose this fun in all seriousness!

Indentured Certitude

raylyons — Wed, 22 Feb 2012 19:20:50 +0000

I want to share some information with you from a resource I mentioned last month. The resource is Edward Suchman’s 1967 book, Evaluative Research and the information is this diagram, which presents a basic model of evaluation:1

I share the diagram because it presents two ideas that don’t always percolate to the top of discussions of library outcome assessment. The first idea is the need for programmatic values to be made explicit beforehand. Suchman, who worked in the public health field, gave this example:

Suppose we begin with the value that it is better for people to have their own teeth rather than false teeth. We may then set our goal that people shall retain their teeth as long as possible.2

Of course, it’s quite possible to hold different values. For instance, one might prefer false teeth over natural ones . . . [Read more]

—————————

1 Suchman, E. A. (1967). Evaluative research: Principles and practice in public service and social action programs, New York: Russell Sage, p.34.
2 Suchman, E. A., p. 35.

The Path of Most Resistance

raylyons — Tue, 31 Jan 2012 16:04:21 +0000

The campaign to assess public library outcomes got a tremendous boost by Library Journal’s Director Summit held last month in Columbus, Ohio. It’s heartening to see library leaders getting serious about making outcome assessment integral to the management of U.S. public libraries! The excitement and determination are necessary for making progress on this front. And it sounds like the summit was designed to let folks absorb relevant ideas in ways that make them their own.

The onset of this newfound energy is the perfect time to commit ourselves to gaining a firm grasp on the core concepts and methods of outcome assessment. Although measurement of outcomes is a new undertaking for libraries, it has been around for a long time in other contexts. In fact, outcome evaluation approaches have been studied, debated, refined, and chronicled over the past forty-five years . . . [Read more]