Paved with Good Intentions

It never hurts to revisit the basics of a method that we’ve chosen to apply to a task we want to accomplish or a problem needing solved. So, the recent announcement of the Library Edge benchmarks is a good occasion to discuss that particular performance assessment method. In the third edition of his book, Municipal Benchmarks, University of North Carolina professor David Ammons describes three types of benchmarking:1

1.  Comparison of performance statistics
2.  Visioning initiatives
3. “Best practices” benchmarking

The idea behind item #1 is that the sufficiency of an organization’s performance can be judged by comparing its performance data with other organizations or against externally defined standards. Comparisons of different organizations using only performance data, without any reference to standards, is called comparative performance measurement. An Urban Institute handbook of the same name by Elaine Morley, Scott Bryant, and Harry Hatry gives an in depth explanation of this method.2 Libraries are already familiar with this approach as comparisons made among selected peers using statistical data for circulation, visits, volumes held, expenditures, registered users, and so on. Library rating systems like the LJ Index of Public Library Service and BIX/Der Bibliotheksindex belong to this category also.

The next type of benchmarking (item #2 above) is visioning initiatives. This approach involves targets set as long-term community goals. These typically are community-wide outcomes meant to represent overall social welfare or quality of life as measured by what are called social indicators. These projects track measures of things such as life expectancy, literacy, crime, civic engagement, and so forth. An example of this approach is the Oregon Benchmarks project.

Best practices benchmarking (item #3 above) comes from the field of quality management. This approach focuses on a specific work process or procedure and studies organizations with outstanding performance records for this process or procedure. The purpose is obtaining ideas for improving the process or procedure in the organization. This is also known as “corporate-style benchmarking” because of its prevalence among private sector organizations. Due to the narrow focus, this approach usually does not reveal much about organizational accomplishments. (Later on I give an example about city street maintenance that illustrates this and should also explain the title I’ve chosen for this post.)

Ammons says that the first benchmarking method—comparison of performance statistics—is the most meaningful for public sector organizations. Comparative statistics provide important context and perspective for an organization’s performance data. Still, as with any tool or method, these have their limitations. For example, consider the fact that statistical comparisons always involve ranking data high to low. The result is that half of the organizations end up classified as higher performers and the other half as lower (using the statistical median to indicate the middle of the group, rather than average). Yet, sometimes the difference between higher and lower performers may be insignificant, especially for organizations that fall near the middle of the distribution. Those slightly below the median are classified as sub-standard while those slightly above are not. It is also possible that the performance of the entire group is unsatisfactory on some organizational measure. However, those at the top of the list still receive high ratings.

This is where standards are useful, including the Library Edge benchmarks (which are essentially standards). Standards can provide an anchor by establishing some minimal level of satisfactory performance. And their content will cover a range of important aspects of a program, service, or intervention. At the same time, the fact that some body or another has issued standards does not automatically make these valid or appropriate. As Ammons observes:

Many so-called [municipal performance] standards are vague or ambiguous, have been developed from limited data or by questionable methods, may focus more on establishing favored policies or practices than on prescribing results, or may be self-serving, more clearly advancing the interests of service providers than service recipients.3

The purpose of comparing performance data—either to other organizations or to externally defined standards—is twofold: (1) to be a catalyst for improving performance and (2) to enable the general public and the elected officials representing them to assess the degree to which a public institution is productive and effective. Although this second purpose is not mentioned in the Library Edge benchmarks, it has been a recognized principle for some time in the library field. The statement below, written in 1956 by historian Gerald W. Johnson and appearing in the Minimum Standards for Public Libraries, 1966, illustrates this:

One obligation resting upon every public institution in a democracy is that of standing ready at all times to render an account of itself to the people and to show cause why they should continue to support it. No institution is so lordly that its right to existence is beyond challenge, and none, except perhaps public monuments, can rightfully claim present consideration on the basis of past distinction.4

For 21st century libraries the closest thing to Johnson’s statement is the catch phrase “demonstrating library value,” which usually means painting as positive a picture of libraries as possible. Nevertheless, Johnson was referring to the notion of accountability that comes from governmental accounting and auditing. The main standard-setting body in that field, the Governmental Accounting Standards Board (GASB), describes it this way:

Public accountability is based on the belief that the taxpayer has a “right to know,” a right to receive openly declared facts that may lead to public debate by the citizens and their elected representatives…Financial reporting should assist in fulfilling government‘s duty to be publicly accountable and should enable users to assess that accountability by… providing information to assist users in assessing the service efforts, costs, and accomplishments of the governmental entity.5

In accountability the main ideas are accuracy, full disclosure, verifiable information, and, as the GASB concept paper notes, assessing accomplishments. This last item is the reason Ammons recommends that standards and benchmarks take the form of “thresholds of effectiveness, service quality, efficiency, outcomes, or results.”6 Citizens evaluate public agencies primarily on accomplishments and benefits delivered at a reasonable cost, rather than on efforts and intentions. Incidentally, best practice benchmarking and continuous improvement regimens don’t really assess accomplishments and benefits. For example, reporting that a city’s street maintenance department uses the most advanced tools and highest quality materials to patch potholes does not inform the public about how responsive and thorough the department is at getting the holes patched, nor how long the patches actually last.

Rather than accomplishments and benefits, the Library Edge benchmarks are about intentions. That is, capacity (sufficient and reliable technology equipment, sufficiently trained staff, user training available, technical assistance offered, relevant website content, organized employment, health, e-government, and education materials, etc.) and service outputs (WiFi/public computer sessions, training and individual assistance delivered, website usage, etc.). This limited scope is appropriate in this early rendition of the benchmarks. But pretty quickly here 21st century libraries are going to need to master more advanced performance measurement methods than just monitoring capacity and outputs.

High quality benchmarks for public library technology services will need to follow the lead of other municipal services. For instance, there are city traffic management departments that evaluate their efforts by measuring improvement in average travel speeds for major traffic arteries during rush hours; and evaluate neighborhood speed-control programs based on measured decreases in traffic speeds and in speeding citations. Public health programs are evaluated on measures like obesity rates and infant mortality rates in the general population. And fire department performance is often gauged not only by elapsed time within which fires are suppressed, but also on the rate of reduction of the incidence of fires community-wide. Of course, these different measures are each compared to data from other towns and cities.

Public library technology proponents should consider developing similar measures beginning with raw counts, for instance, numbers of e-government transactions, vocational exams, or online education courses successfully completed by patrons. Then, to conduct true benchmarking these counts need to be made comparable across libraries and communities (as Ammons advises). That is, they need computed as rates, such as number of e-government transactions successfully completed per public computer or WiFi session, or per e-government training session hour offered; or percentage of e-government training class attendees who test competent in the use of selected e-government sites.

When public library technology benchmarking progresses to this stage, then our field will be leaders rather than followers in performance measurement. We will also have extraordinarily meaningful data to report to library boards, elected officials, and taxpayers. Quite a shift from reporting routine counts of this year’s service outputs, bandwidth increases, or even splashy new technologies installed. And, by the way, with measures of accomplishments and benefits the Library Edge value-of-public-computing stories will document the actual destinations public technology leads to.

 
—————————

1   Ammons, D. N. (2012). Municipal benchmarks: Assessing local performance and establishing community standards, Armonk, NY: M.E. Sharpe, p. 15.
2  Morley, E., Bryant, S. P., & Hatry, H. P. (2001). Comparative performance measurement, Washington, DC: The Urban Institute Press.
3  Ammons, D. N. (2012). p. 4.
4  American Library Association. (1967). Standards for Public Library Systems, 1966,, Chicago: American Library Association, p. 1.
5  Governmental Accounting Standards Board. (1987). Concept Statement 1: Objectives of Financial Reporting, p. ii. Retrieved from http://www.gasb.org/jsp/GASB/Page/GASBSectionPage&cid=1176160042391#gasbcs5.
6  Ammons, D. N. (2012). p. 16.

3 thoughts on “Paved with Good Intentions

  1. I’ve read a study where an academic library tracked first year GPAs but thinking of what a public library could track leaves me wondering if we do anything useful. Literacy rates were the first thing that comes to mind but does the public library claim to increase literacy? The library I work at has been given the mandate to work on early childhood literacy and provide a meeting space for the community. Should we be looking at Grade 1 test scores? or local crime?
    I’ve been scratching my head over a useful metric to use to evaluate the summer reading program I am designing because I agree participant numbers and programming hours already presuppose that you think the the program is worthwhile. And the general claim that summer reading programs prevent students from forgetting over the summer what they learned is backed by very thin and shaky research.

    1. Thanks for your comment, Carla. Yes, determining effectiveness is a daunting task and we have to do as well as we can given the resources available. I have seen some discussion of academic libraries looking at GPA’s as being too narrow a success criterion. Helping students graduate (with whatever GPA they happen to earn) may well be the goal that libraries should be contributing to. For summer reading you might try a dual-approach. First, use materials and methods whose quality/effectiveness have been proven somehow (do these exist?). Second, give reading proficiency pre-tests and post-tests to summer program attendees. This won’t “experimentally” confirm that your summer program aided proficiency at the end of the program (by avoiding proficiency loss or leading to gains), but at least it will be apparent if/how attendees progressed.

      Being unfamiliar with this area, let me ask if there are other legitimate goals to be considered? E.g. could success be an increase in the amount of time children voluntarily spend reading at home? Is that a useful indicator? Or is getting children to prefer a wider range of story books, rather than reading the same books repeatedly? (Of course, that presumes variety contributes to reading skills–does it? Or is repetition more important?)

      And would an increase in numbers of children’s books owned at home be a useful indicator? I’m just guessing here, looking for success indicators beyond maintaining/improving tested reading proficiency. Do you think there are other positive results that might bolster reading that are not necessarily reflected in proficiency scores? Say for children who don’t take that well to reading? Any obstacles that get removed would seem to pave the way for future reading improvement.

      On that topic, of course, it is recognized that summer reading programs tend to attract the more proficient students, those already comfortable with reading. Students with more difficulties, either in reading or in family issues at home, tend not to attend or drop out mid-course. So, the issue of outreach to the most needy students is a big one, which I suppose you knew already.

  2. Interesting post, Simple metrics like doorcounts and circulation only tell part of the story, and situations like having toilets outside the library can inflate door statistics and changes to circulation policy such as loan periods may reduce or inflate statistics.
    In academic libraries there is a lot of increase in measuring the value of the library in terms of student outcomes, but this is not easy to do.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s