It never hurts to revisit the basics of a method that we’ve chosen to apply to a task we want to accomplish or a problem needing solved. So, the recent announcement of the Library Edge benchmarks is a good occasion to discuss that particular performance assessment method. In the third edition of his book, Municipal Benchmarks, University of North Carolina professor David Ammons describes three types of benchmarking:1
1. Comparison of performance statistics
2. Visioning initiatives
3. “Best practices” benchmarking
The idea behind item #1 is that the sufficiency of an organization’s performance can be judged by comparing its performance data with other organizations or against externally defined standards. Comparisons of different organizations using only performance data, without any reference to standards, is called comparative performance measurement. An Urban Institute handbook of the same name by Elaine Morley, Scott Bryant, and Harry Hatry gives an in depth explanation of this method.2 Libraries are already familiar with this approach as comparisons made among selected peers using statistical data for circulation, visits, volumes held, expenditures, registered users, and so on. Library rating systems like the LJ Index of Public Library Service and BIX/Der Bibliotheksindex belong to this category also.
The next type of benchmarking (item #2 above) is visioning initiatives. This approach involves targets set as long-term community goals. These typically are community-wide outcomes meant to represent overall social welfare or quality of life as measured by what are called social indicators. These projects track measures of things such as life expectancy, literacy, crime, civic engagement, and so forth. An example of this approach is the Oregon Benchmarks project.
Best practices benchmarking (item #3 above) comes from the field of quality management. This approach focuses on a specific work process or procedure and studies organizations with outstanding performance records for this process or procedure. The purpose is obtaining ideas for improving the process or procedure in the organization. This is also known as “corporate-style benchmarking” because of its prevalence among private sector organizations. Due to the narrow focus, this approach usually does not reveal much about organizational accomplishments. (Later on I give an example about city street maintenance that illustrates this and should also explain the title I’ve chosen for this post.)
Ammons says that the first benchmarking method—comparison of performance statistics—is the most meaningful for public sector organizations. Comparative statistics provide important context and perspective for an organization’s performance data. Still, as with any tool or method, these have their limitations. For example, consider the fact that statistical comparisons always involve ranking data high to low. The result is that half of the organizations end up classified as higher performers and the other half as lower (using the statistical median to indicate the middle of the group, rather than average). Yet, sometimes the difference between higher and lower performers may be insignificant, especially for organizations that fall near the middle of the distribution. Those slightly below the median are classified as sub-standard while those slightly above are not. It is also possible that the performance of the entire group is unsatisfactory on some organizational measure. However, those at the top of the list still receive high ratings.
This is where standards are useful, including the Library Edge benchmarks (which are essentially standards). Standards can provide an anchor by establishing some minimal level of satisfactory performance. And their content will cover a range of important aspects of a program, service, or intervention. At the same time, the fact that some body or another has issued standards does not automatically make these valid or appropriate. As Ammons observes:
Many so-called [municipal performance] standards are vague or ambiguous, have been developed from limited data or by questionable methods, may focus more on establishing favored policies or practices than on prescribing results, or may be self-serving, more clearly advancing the interests of service providers than service recipients.3
The purpose of comparing performance data—either to other organizations or to externally defined standards—is twofold: (1) to be a catalyst for improving performance and (2) to enable the general public and the elected officials representing them to assess the degree to which a public institution is productive and effective. Although this second purpose is not mentioned in the Library Edge benchmarks, it has been a recognized principle for some time in the library field. The statement below, written in 1956 by historian Gerald W. Johnson and appearing in the Minimum Standards for Public Libraries, 1966, illustrates this:
One obligation resting upon every public institution in a democracy is that of standing ready at all times to render an account of itself to the people and to show cause why they should continue to support it. No institution is so lordly that its right to existence is beyond challenge, and none, except perhaps public monuments, can rightfully claim present consideration on the basis of past distinction.4
For 21st century libraries the closest thing to Johnson’s statement is the catch phrase “demonstrating library value,” which usually means painting as positive a picture of libraries as possible. Nevertheless, Johnson was referring to the notion of accountability that comes from governmental accounting and auditing. The main standard-setting body in that field, the Governmental Accounting Standards Board (GASB), describes it this way:
Public accountability is based on the belief that the taxpayer has a “right to know,” a right to receive openly declared facts that may lead to public debate by the citizens and their elected representatives…Financial reporting should assist in fulfilling government‘s duty to be publicly accountable and should enable users to assess that accountability by… providing information to assist users in assessing the service efforts, costs, and accomplishments of the governmental entity.5
In accountability the main ideas are accuracy, full disclosure, verifiable information, and, as the GASB concept paper notes, assessing accomplishments. This last item is the reason Ammons recommends that standards and benchmarks take the form of “thresholds of effectiveness, service quality, efficiency, outcomes, or results.”6 Citizens evaluate public agencies primarily on accomplishments and benefits delivered at a reasonable cost, rather than on efforts and intentions. Incidentally, best practice benchmarking and continuous improvement regimens don’t really assess accomplishments and benefits. For example, reporting that a city’s street maintenance department uses the most advanced tools and highest quality materials to patch potholes does not inform the public about how responsive and thorough the department is at getting the holes patched, nor how long the patches actually last.
Rather than accomplishments and benefits, the Library Edge benchmarks are about intentions. That is, capacity (sufficient and reliable technology equipment, sufficiently trained staff, user training available, technical assistance offered, relevant website content, organized employment, health, e-government, and education materials, etc.) and service outputs (WiFi/public computer sessions, training and individual assistance delivered, website usage, etc.). This limited scope is appropriate in this early rendition of the benchmarks. But pretty quickly here 21st century libraries are going to need to master more advanced performance measurement methods than just monitoring capacity and outputs.
High quality benchmarks for public library technology services will need to follow the lead of other municipal services. For instance, there are city traffic management departments that evaluate their efforts by measuring improvement in average travel speeds for major traffic arteries during rush hours; and evaluate neighborhood speed-control programs based on measured decreases in traffic speeds and in speeding citations. Public health programs are evaluated on measures like obesity rates and infant mortality rates in the general population. And fire department performance is often gauged not only by elapsed time within which fires are suppressed, but also on the rate of reduction of the incidence of fires community-wide. Of course, these different measures are each compared to data from other towns and cities.
Public library technology proponents should consider developing similar measures beginning with raw counts, for instance, numbers of e-government transactions, vocational exams, or online education courses successfully completed by patrons. Then, to conduct true benchmarking these counts need to be made comparable across libraries and communities (as Ammons advises). That is, they need computed as rates, such as number of e-government transactions successfully completed per public computer or WiFi session, or per e-government training session hour offered; or percentage of e-government training class attendees who test competent in the use of selected e-government sites.
When public library technology benchmarking progresses to this stage, then our field will be leaders rather than followers in performance measurement. We will also have extraordinarily meaningful data to report to library boards, elected officials, and taxpayers. Quite a shift from reporting routine counts of this year’s service outputs, bandwidth increases, or even splashy new technologies installed. And, by the way, with measures of accomplishments and benefits the Library Edge value-of-public-computing stories will document the actual destinations public technology leads to.
1 Ammons, D. N. (2012). Municipal benchmarks: Assessing local performance and establishing community standards, Armonk, NY: M.E. Sharpe, p. 15.
2 Morley, E., Bryant, S. P., & Hatry, H. P. (2001). Comparative performance measurement, Washington, DC: The Urban Institute Press.
3 Ammons, D. N. (2012). p. 4.
4 American Library Association. (1967). Standards for Public Library Systems, 1966,, Chicago: American Library Association, p. 1.
5 Governmental Accounting Standards Board. (1987). Concept Statement 1: Objectives of Financial Reporting, p. ii. Retrieved from http://www.gasb.org/jsp/GASB/Page/GASBSectionPage&cid=1176160042391#gasbcs5.
6 Ammons, D. N. (2012). p. 16.