Lib(rary) Performance Blog

I think I get it now.  I had thought the term assessment meant a systematic and appropriately rigorous measurement of a construct or phenomenon of interest, like program outcomes, community needs, service quality, and so on.  Only now have I come to understand that a self-assessment is a different animal altogether. Who would have thought that the purpose of a self-assessment is not really to assess anything?  The purpose, I now realize, is to inform and educate. All this time I have been applying research methodology standards to tools that are intended to advocate and indoctrinate. No wonder my observations have been so off-base!

When I disapproved of WebJunction’s online competencies assessment questionnaire (see my April 22, 2009 entry), the WebJunction staff explained to me that the true objective for their surveys was to increase awareness of these competencies. I immediately wondered, “Well, how then will WebJunction measure awareness?”  But that is quite an irrelevant question when these questionnaires are actually teaching tools, not measurement instruments. Since the instruments don’t really have to measure anything, we don’t have to obsess about how reliable or valid they are. They can be evaluated (I guess) according to how well they apply proven methods for facilitating adult learning.

The irony of using a research instrument like a survey questionnaire this way will probably escape the majority of librarians (i.e. those who disliked library school research methods class.)  But here’s the story: One of the giant problems in designing behavioral science measures is making sure the measures don’t alter the thing you’re trying to measure. Measures are supposed to be unobtrusive. You would never trust a thermometer if you found that, while measuring the temperature of water, the thermometer also happened to heat the water! The same goes for questionnaires and tests in behavioral science and education.

Worries like this are old hat nowadays. Forget the antiseptic, hands-off approach. So easy and cheap to post online, the new questionnaires are designed to induce change by informing, educating, and motivating respondents. I ran across another one of these in connection with a new initiative on “21st century skills” launched last week by the Institute of Museum and Library Services (IMLS). This campaign presents a thoroughly modern take on the mission of libraries and museums. You can read the details and access the “self-assessment tool” here.

Still stuck in my 20th century research methodology paradigm, I found the IMLS questionnaire technically interesting. It is what I call a “Goldilocks instrument” since it uses a 3-point ordinal scale that amounts to a little, a medium amount, and a lot. The response options are something like this:

1. The institution rarely practices such-and-such 21st century skills enhancement task or technique
2. The institution practices the task or technique fairly often, or
3. The institution almost always practices the task or technique.

In several questions in the survey, this tripartite scale appears as less than 25% of the time, 25% to 75% of the time, and over 75% of the time. But you get the idea—small, medium, large.

Specific questionnaire items address a series of general institutional dimensions like accountability, leadership, partnerships, and so on.  (See the self-assessment tool matrix.)  Then, in each area, the institution is rated as being in one of three developmental stages:  Early, Transitional, or 21st Century. An institution’s Goldilocks responses fall conveniently into these stages (surprise!!).  If you perform a 21st century skill enhancement task less than 25% of the time, you are in the Early (Neolithic?) stage on that one.  If you perform it more than 75% of the time, you are thoroughly modern!

At the completion of the questionnaire, the self-assessment tool simply parrots back an institution’s responses in graphical form. There are “Recommendations” buttons users can click on, but the advice offered is pretty much the same, regardless of an institution’s rating: Use the results “to initiate a dialogue with your institution’s leaders, board, colleagues, and other stakeholders” so you can improve your rating. In Goldilocks measurement terms, having the most 21st century skills possible is always “just right!”

Obviously, the survey is a teaching tool, not an assessment. That’s why there is no need for the instrument to gauge how libraries and museums compare to any independently derived standards.  Like some “minimum recommended daily allowance” of a particular 21st century practice. This makes things much simpler for IMLS because the idea of library or museum standards, itself, is notoriously tricky.  Several of the approaches endorsed in their model don’t apply to many institutions.  (How can a small rural library or a historic police museum be collaborating with community partners on its new educational programs “over 75% of the time?”)

Fortunately, these types of measurement issues are immaterial.  Remember, this is not assessment.  It is education and proselytizing.  In fact, the IMLS self-assessment tool demonstrates one 21st century skill enhancement technique first-hand. As described in the project report, the tool is clearly interactive audience involvement! Rather than posting the questionnaire merely to measure something, IMLS is modeling the behavior they are seeking from museums and libraries.  I think it’s called “showing by doing.”

I noticed that yet another library value calculator has appeared on the scene. This one is offered by the National Network of Libraries of Medicine (NNLM) with the very best of intentions, I am sure. But, let me say that I am convinced that these calculators are a bad idea. Their underlying assumptions are weak and their designs are not well thought out. Eventually, library funders and stakeholders are going to realize that the calculations are superficial and…well…sloppy.

For one thing, sound cost-benefit analysis requires an examination of the full extent of relevant costs and benefits of a given project, program, or service. These quick-and-easy library calculators, however, use average retail prices as proxies for benefits. This oversimplification ignores important sources of library value like contributions to student and life-long learning, scientific and academic research, and public discourse, as well as roles libraries play in imparting cultural and humanitarian values and traditions, promoting literary appreciation and aesthetic values, facilitating community cohesion, and so forth.

But say that, for practical purposes, we accept the idea that value-boils-down-to-price as reasonable. Even so, the retail pricing approach these calculators use has definite problems. The calculators view retail prices as estimates of costs that patrons would incur if the library’s items and services were—hypothetically—unavailable to the community or institution. The library comes up with a retail price for each type of material and service it offers, and then these prices are translated directly into the value patrons receive from utilizing these materials or services.

In many cases, however, the alternative to obtaining an item or service from the library is not an outright purchase at retail prices. A student might purchase a textbook for $125 and then later re-sell it on Amazon.com for $50. Or perhaps she buys the item at a used price or borrows it from a friend for free. Clearly, a variety of alternative patron scenarios are possible, meaning that there is a range of alternative costs (approximate values) associated with each item or service use. The average of these ranges will typically be less than an item’s retail price. Besides, an item borrowed from a library does not include the breadth of rights and conveniences that item ownership does. So, it is a stretch to say that a patron always enjoys the same benefit from a borrowed item as from a purchased one.

Other problems with the calculators make their output suspect. For example, each time a patron renews an item or re-uses it in-house or online, the item’s retail price gets credited—again—to the library’s value totals. (Cha-ching!) On the other hand, when our Amazon.com shopper purchases a book at $75, that book’s value does not increase to $150, then to $225 and beyond each time the owner opens the book, or with each 3-week library loan period that passes.

Because the calculators tally only certain types of transactions, they end up painting a rather rosy picture of library performance. Consider the case of a patron who needs an item or service that is (really) not available from the library, and whose information need ultimately goes unmet. And the case of a service delivered that fails to meet a patron’s need, such as an unproductive reference consultation. The first case won’t be tallied at all by these calculators, and the second case will be tallied but will be significantly over-valued. (It will be considered a complete success.) Yet, the actual value of both of these patron transactions is negative and should be entered into these calculators this way. Unfortunately, the calculators’ designs do not accommodate this.

Given these problems and oversights, it is fairly obvious that these calculators produce exaggerated estimates of the benefits which libraries provide. Perhaps this exaggeration is only moderate or perhaps it is substantial—we cannot really tell for sure.

The calculators also underestimate the cost side of the equations, causing their benefit/cost ratios to be even more over-stated. They ignore several key costs incurred in delivering library materials and services, including expenses for information technology, equipment, building maintenance, utilities, and administrative overhead. These calculators also disregard the incidental costs that patrons may bear, like travel and parking costs, time lost due to item unavailability or poor service, usability difficulties encountered, and so on. In fact, NNLM’s calculator errors in the opposite direction: Assuming that libraries are always convenient, the calculator builds a patron time-savings factor into its formula. (I suppose you could enter in negative numbers to register patron lost time and inconvenience.)

When the calculators do recognize costs, they end up settling for data that are the grossest of estimates. For instance, users can enter estimated percent of total library staff time spent supporting access to materials or services. Creators of the calculators seem unaware that accurate benefit/cost ratios require meticulous collection of operational data, not just convenient guess-timates.

You will be hard-pressed to learn about these shortcomings from the materials that accompany library value calculators. Mostly, libraries receive general guidelines for entering data and encouragement to use the calculators without reservation. The library just keys in its data and—voilà!—receives an exact return-on-investment percentage or benefit/cost ratio right on the spot! Given the casual assumptions the calculations entail and the inexactness of the library’s input data, you’d think the final answer would at least include some type of margin-of-error disclaimer. Maybe something like this:

Your library’s benefit-to-cost ratio = $8.20 per $1.00 cost*

*    Based on our calculations, we are 95% confident that your library’s benefit/cost ratio is between $4.50 and $12.50 (per $1.00 cost). If your data are especially inaccurate, this range will be larger. Note that our single $8.20 estimate may be high due to assumptions our model uses.

Needless to say, this kind of small print doesn’t appear in the instructions that come with library value calculators. As they are, the calculators generate figures that are precise to the penny, with no other explanations to speak of. Libraries confidently report the figures to stakeholders as accurate, authoritative, and nearly approaching Scientific Truth. Of course, the figures are nothing of the sort.

Clones of these library calculators have sprouted up on dozens of library websites, where patrons are invited to enter their custom data to receive their own monthly “value of library services.” Costs are typically not mentioned, so that final value calculations are simple multiplications of counts times arbitrary and often fanciful retail price estimates. Of course, the nifty and optimistic totals will delight library patrons. The totals might even please the population of nonusers who are happy to subsidize library use by others as an overall benefit to the community or institution.

On a few public library websites the calculations are made even more tantalizing by informing patrons about their “individual return-on-investment”—how much value they gain for every tax dollar they contribute. (Don’t you just love democracy!) Unfortunately, this approach casts the wrong light on the public value of libraries. First, the figures are further exaggerations because they use per capita revenue data. Not every public library user pays taxes, a fact that makes the individually quoted return rates artificially high. (Instead of library tax revenue per capita, the calculations should use tax revenue per tax-paying household.)

Second, these seemingly benign “value” calculations actually hide information. The websites fail to provide overall return-on-investment rates for all tax-paying households or for all tuition-paying students. As I have already alluded, an individual patron’s rate of return is being subsidized by nonusers of library services. For every patron elated with his own personally-calculated rate, there will be several households or students whose return rates are less than $0, meaning they lose money on their library tax or tuition “investments.” (This mix of returns rates also applies to using the vanilla versions of the calculators that don’t bother to factor costs in.) Omitting this larger picture from these presentations is slanted and misleading—something that libraries should not be involved in.

From a public or institutional value perspective, these Library 2.0-inspired patron calculators completely sidestep the rightful purpose of library evaluation. This purpose is to assess the extent to which the library provides value to the institution or community as a whole, not how each individual fares. This assessment must also confirm that products and services are equitably distributed, that is, equally available and accessible to all who wish or need to use them (see Creating Public Value by Mark H. Moore).

In actuality, economic valuation is not so simple as it appears. It involves complicated (and frustrating) concepts like exchange value, use value, contingent value, and others. Even business corporations have misgivings about standard return-on-investment analysis because of how difficult it is to obtain reliable data to input into the formulas.

If we want to use purely monetary estimates of the value of our services, we need more rigorous methods than these makeshift library calculators. This exact advice was offered to us a couple of years ago by Donald Elliot and Glen and Leslie Holt in their book Measuring Your Library’s Value. Their work provides important guidance that we should be heeding. Like the fact that benefit/cost valuations are unique to the communities and institutions from whence they come. The figures are really not comparable across communities or for different libraries. This is something that most of us would not have thought about. The central message from their book, though, should already be obvious to us: We can’t just make these benefit/cost numbers up, the way these calculators do. There have to be sound theoretical and empirical bases for our findings.

Sure, quick-and-dirty estimations might be helpful in certain situations, as long as they are recognized for what they are. But the numbers gushing from these library calculators are nonsensical and disingenuous, in many cases. The whole idea has become an impediment to the real work of assessing library value. When the batteries in these little pocket library calculators wear out, I recommend that we just not replace them.