Feeds:
Posts
Comments

Data Are Not Psychic

November 28, 2011 by raylyons


It’s great to see other librarians advocating for the same causes I harp on in this blog. I’m referring to Sarah Robbins, Debra Engel, and Christina Kulp of the University of Oklahoma, whose article appears in the current issue of College & Research Libraries. The article, entitled “How Unique Are Our Users?”1  warns against the folly of using convenience samples. It implores library researchers to honestly explain the limitations of their studies. And the authors are resolute about the importance of understanding the generalizability of survey findings, a topic which also happens to be the main focus of their study.

I bring up their article for a different reason, however. It is an example of how difficult and nuanced certain aspects of research and statistics can be. Despite the best of intentions, it’s amazingly easy to get tripped up by one or another detail. Robbins and her colleagues got caught in the briar patch that is statistics and research methods. I say so because the main conclusions reached in their study are not actually borne out by their survey results.     [Read more...]

  
—————————

1  Robbins, S., Engel, D. and Kulp, C., 2011, “How Unique Are Our Users? Comparing Responses Regarding the Information-Seeking Habits of Engineering Faculty,” College & Research Libraries, 72:6, pp. 515-532.

Posted in Measurement, Research, Statistics | Tagged , , , , , | Leave a Comment »

Beauty Is As Beauty Does

October 28, 2011 by raylyons


Infographics is one of two new fashionable terms used nowadays to refer to statistical charts and graphs. The other term is visualizations, which replaces such archaic words as graphs, charts, pictures, diagrams, and illustrations. Sometimes the term is affectionately shortened to data viz by its really cool practitioners.

In the infographics/visualization/data viz movement there are two basic schools of thought. One school emphasizes principles of artistic design and the other emphasizes information clarity. The first prizes graphics that are beautiful and appealing, while the other judges visualizations based on how informative they are.1  Many adherents of the first approach to graphics are marketing and advertising professionals. Lest you presume that they subscribe to the motto
ars gratia artis…     [Read more...]

 
—————————

1  Of course, it is possible for graphics to be simultaneously beautiful and informational. Well-designed graphics can be elegant in their clarity and visual appeal. See Edward Tufte’s book Beautiful Evidence.

Posted in Uncategorized | Leave a Comment »

Library Science

September 29, 2011 by raylyons


Evaluation, assessment, and performance measurement are not what you’d call sciences. But these activities do share certain things in common with science and the scientific method.1  One is the requirement that theories be tested based on the compilation of objective evidence. Another is the idea of replication, which is carefully repeating a measurement or experiment in order to verify that the initial findings were not an accident or mistake of some sort.

Then there’s the more philosophical concept known as falsifiablity. A scientific theory needs to be such that there is some way that it can be examined and possibly disproved. A credible scientific theory is one that holds up under repeated attempts to be proven wrong.

In everyday terms, there is a lot of transparency and double-checking in science. I bring these ideas up because, as it happens, there is a claim made in my prior blog entry that needs rechecked. The claim is:

On the basis of per capita statistics, smaller U.S. public libraries out-perform the largest U.S. public libraries.

[Read more...]

—————————

1  Some of the foundational ideas in evaluation, assessment, and especially performance measurement have also been borrowed from the field of financial auditing. See Beryl Radin’s 2006 book, Challenging the Performance Movement: Accountability, Complexity, and Democratic Values and Michael Power’s 1997 book, The Audit Society: Rituals of Verification.

Posted in Measurement, Reporting Evaluation/Assessment Results, Research | Tagged , | Leave a Comment »

How Do You Know That?

August 23, 2011 by raylyons

I borrowed the title for this entry from a 2009 study of student research practices by Randall McClure and Kellian Clink. Their study is cited in an article in the current issue of College & Research Libraries that Joe Matthews brought to my attention. This article is Students Use More Books After Library Instruction by Rachel Cooke and Danielle Rosenthal. Both articles explore research sources and citations that undergraduate students use in writing assignments. Though it’s the second article I want to discuss, McClure’s and Clink’s well-chosen title is too good to pass up. In fact, I’m thinking of making it the motto of this blog!

Anyway, in their article Cooke and Rosenthal report that university English composition students “used more books, more types of sources, and more overall sources when a librarian provided instruction.”1  Their statement contains two separate claims…   [Read more...]

 
—————————

1   Cooke, R. and Rosenthal, D., 2011, Students Use More Books after Library Instruction: An Analysis of Undergraduate Paper Citations, College & Research Libraries, 72:4, p. 332.

Posted in Measurement, Research, Statistics | Tagged , , , , , | Leave a Comment »

Beware of Vengeful Prayer

July 21, 2011 by raylyons

I recently ran across a series of studies suggesting that prayer tends to lessen anger and aggression. Researchers concluded that prayer helps people adopt a more positive view of adverse or irritating circumstances. There also happens to be a sideline to their findings that illustrates something you don’t hear much about from proponents of outcomes assessment in libraries. It involves this statement by the researchers:

These results would only apply to the typical benevolent prayers that are advocated by most religions… Vengeful or hateful prayers, rather than changing how people view a negative situation, may actually fuel anger and aggression.

Though the aims of the prayer studies differ from those of library outcomes studies, the two research approaches are similar in this respect: When studying effects of a program, treatment, or intervention, if we’re not sure about the exact content of that program, treatment, or intervention, then we have a problem. In the field of program evaluation this problem falls under the rubric of program fidelity.   [Read more...]

Posted in Accountability, Outcome assessment, Program implementation | Leave a Comment »

Smaller is Beautifuler

June 17, 2011 by raylyons

A recent article in AL Direct entitled The Smartest Readers presents some simple library rankings based on that stalwart library measure, circulation per capita. Rankings like these are, at least to me, a reminder of a perennial conundrum concerning the meaning of per capita library measures. For more than a century librarianship has puzzled over how to evaluate these statistics. Do per capita data tell us whether or not libraries are doing a good job? What amounts of materials made available or levels of services delivered are sufficient for libraries with specific missions and serving communities of a particular size and makeup?

Mainly, libraries have to rely on their own ingenuity to interpret per capita or per constituent data (like per student, faculty, employee, subscriber, stakeholder, and such). About the only official guidance they have gotten over the decades is advice about comparing (benchmarking) their data with appropriate peer libraries. Lacking some more objective gauge of statistical performance, libraries end up applying what might be called the more-is-better rule.   [Read more...]

Posted in Library assessment, Measurement, Statistics | Leave a Comment »

Fickle Users of Figures

May 11, 2011 by raylyons

The field of program evaluation has grappled with the political context of institutional performance measurement for decades. For libraries and universities, though, the politics of accountability is newer terrain. In some instances these organizations have unwittingly enrolled in a crash course on the subject, learning in real-time how volatile the process can be.

A prime example is the recent controversy about faculty productivity within the University of Texas System (UT). At the request of the its Board of Regents, UT released a 821 page spreadsheet disclosing detailed records on faculty compensation, course enrollment, class sections taught, research time allocations, and other related data. Each page of the document contains this curious disclaimer in red:

The data in its current draft form is incomplete and has not yet been fully verified or cross referenced. In its present raw form it cannot yield accurate analysis, interpretations or conclusions.

Essentially, they’re saying, “Well, here are our data, but they can’t be trusted.” [Read more...]

Posted in Accountability, Reporting Evaluation/Assessment Results, Statistics | Tagged , , | Leave a Comment »

The U Word

April 9, 2011 by raylyons

This week Chase Bank sent an email to its customers saying that one of their vendor’s computer systems were hacked. The bank stated that they:

…are confident that the information that was retrieved [i.e., stolen] included some Chase customer e-mail addresses, but did not include any customer account or financial information. Based on everything we know, your accounts and financial information remain secure.

Confidence based on whatever they happen to know, eh?  Because Chase could easily be mistaken, customers would be foolish to put their full trust in the bank’s assurances. I definitely plan to keep an eye on my Chase account for the next several months.

This same caution also applies to the most recent OCLC membership report, Perceptions of Libraries, 2010: Context and Community. The report’s energetic graphics and narrative make the information seem to be true. But, as my prior posts1 explain, surveys are always incomplete and imperfect. Findings from a single survey like OCLC’s are just not weighty enough to deserve our unconditional trust…    [Read more...]

 
—————————

1   See Discussing Accuracy, Checking It Twice, Stranger Than Fiction, and Objects In Mirror Are Closer Than They Appear.

Posted in Advocacy, Measurement, Reporting Evaluation/Assessment Results, Research, Statistics | Leave a Comment »

Stubborn Facts

March 12, 2011 by raylyons

In the book John Adams author David McCullough writes about Adams’ legal defense of British soldiers on trial for murder in 1770. In his argument to the Massachusettes jury Adams said:

Facts are stubborn things. And whatever our wishes, our inclinations, or the dictums of our passions, they cannot alter the state of facts and evidence.1

Indisputable facts are difficult to ignore, indeed. Yet, facts are not always clear and unambiguous. Getting to the plain facts and drawing valid conclusions from them can be stubborn matters in their own right. To quote science teacher and YouTube lecturer, wonderingmind42, “Interpreting evidence well requires skill, training, and experience.”2    [Read more...]

 
—————————

1  In McCullough, D., 2001, John Adams, Simon & Schuster, p. 68.  Red emphasis added.
2  Quote appears in the video at the 5:18 time mark. Also watch the segment from 2:40 to 4:20 about facts versus the interpretation of facts.

Posted in Advocacy, Reporting Evaluation/Assessment Results, Research | Leave a Comment »

Non-Exponential Potential

February 15, 2011 by raylyons

A new OCLC membership report, Perceptions of Libraries, 2010: Context and Community, is hot off the…er…PDF-Maker! The report is formatted more like a magazine than a study, with key findings summarized in a myriad of graphical illustrations. So, I must confess that I have rather neglected the narrative so far. But from browsing mostly through the pictures, I have come up with a few suggestions that might enhance the report’s message, quantitatively speaking.

First, it would be better if the OCLC market researchers avoided citing large percentages, like the 1,544% growth in e-Book sales (p. 11) and 1,050% growth in smart phone ownership (p. 15). As Derrick Niederman and David Boyum explain in their book, percentages like these tend to be overstatements due to the baseline figures used. And the percentages just aren’t that informative …  [Read more...]

Posted in Advocacy, Measurement, Reporting Evaluation/Assessment Results | Leave a Comment »

Older Posts »

Follow

Get every new post delivered to your Inbox.