Lib(rary) Performance » Research

Data Are Not Psychic

raylyons — Mon, 28 Nov 2011 19:30:33 +0000

It’s great to see other librarians advocating for the same causes I harp on in this blog. I’m referring to Sarah Robbins, Debra Engel, and Christina Kulp of the University of Oklahoma, whose article appears in the current issue of College & Research Libraries. The article, entitled “How Unique Are Our Users?”1 warns against the folly of using convenience samples. It implores library researchers to honestly explain the limitations of their studies. And the authors are resolute about the importance of understanding the generalizability of survey findings, a topic which also happens to be the main focus of their study.

I bring up their article for a different reason, however. It is an example of how difficult and nuanced certain aspects of research and statistics can be. Despite the best of intentions, it’s amazingly easy to get tripped up by one or another detail. Robbins and her colleagues got caught in the briar patch that is statistics and research methods. I say so because the main conclusions reached in their study are not actually borne out by their survey results. [Read more...]

—————————

1 Robbins, S., Engel, D. and Kulp, C., 2011, How unique are our users? Comparing responses regarding the information-seeking habits of engineering faculty, College & Research Libraries, 72:6, pp. 515-532.

Library Science

raylyons — Thu, 29 Sep 2011 19:46:38 +0000

Evaluation, assessment, and performance measurement are not what you’d call sciences. But these activities do share certain things in common with science and the scientific method.1 One is the requirement that theories be tested based on the compilation of objective evidence. Another is the idea of replication, which is carefully repeating a measurement or experiment in order to verify that the initial findings were not an accident or mistake of some sort.

Then there’s the more philosophical concept known as falsifiablity. A scientific theory needs to be such that there is some way that it can be examined and possibly disproved. A credible scientific theory is one that holds up under repeated attempts to be proven wrong.

In everyday terms, there is a lot of transparency and double-checking in science. I bring these ideas up because, as it happens, there is a claim made in my prior blog entry that needs rechecked. The claim is:

On the basis of per capita statistics, smaller U.S. public libraries out-perform the largest U.S. public libraries.

[Read more...]

—————————

1 Some of the foundational ideas in evaluation, assessment, and especially performance measurement have also been borrowed from the field of financial auditing. See Beryl Radin’s 2006 book, Challenging the Performance Movement: Accountability, Complexity, and Democratic Values and Michael Power’s 1997 book, The Audit Society: Rituals of Verification.

How Do You Know That?

raylyons — Tue, 23 Aug 2011 15:32:57 +0000

I borrowed the title for this entry from a 2009 study of student research practices by Randall McClure and Kellian Clink. Their study is cited in an article in the current issue of College & Research Libraries that Joe Matthews brought to my attention. This article is Students Use More Books After Library Instruction by Rachel Cooke and Danielle Rosenthal. Both articles explore research sources and citations that undergraduate students use in writing assignments. Though it’s the second article I want to discuss, McClure’s and Clink’s well-chosen title is too good to pass up. In fact, I’m thinking of making it the motto of this blog!

Anyway, in their article Cooke and Rosenthal report that university English composition students “used more books, more types of sources, and more overall sources when a librarian provided instruction.”1 Their statement contains two separate claims… [Read more...]

—————————

1 Cooke, R. and Rosenthal, D., 2011, Students Use More Books after Library Instruction: An Analysis of Undergraduate Paper Citations, College & Research Libraries, 72:4, p. 332.

The U Word

raylyons — Sun, 10 Apr 2011 00:50:54 +0000

This week Chase Bank sent an email to its customers saying that one of their vendor’s computer systems were hacked. The bank stated that they:

…are confident that the information that was retrieved [i.e., stolen] included some Chase customer e-mail addresses, but did not include any customer account or financial information. Based on everything we know, your accounts and financial information remain secure.

Confidence based on whatever they happen to know, eh? Because Chase could easily be mistaken, customers would be foolish to put their full trust in the bank’s assurances. I definitely plan to keep an eye on my Chase account for the next several months.

This same caution also applies to the most recent OCLC membership report, Perceptions of Libraries, 2010: Context and Community. The report’s energetic graphics and narrative make the information seem to be true. But, as my prior posts1 explain, surveys are always incomplete and imperfect. Findings from a single survey like OCLC’s are just not weighty enough to deserve our unconditional trust… [Read more...]

—————————

1 See Discussing Accuracy, Checking It Twice, Stranger Than Fiction, and Objects In Mirror Are Closer Than They Appear.

Stubborn Facts

raylyons — Sat, 12 Mar 2011 03:55:02 +0000

In the book John Adams author David McCullough writes about Adams’ legal defense of British soldiers on trial for murder in 1770. In his argument to the Massachusettes jury Adams said:

Facts are stubborn things. And whatever our wishes, our inclinations, or the dictums of our passions, they cannot alter the state of facts and evidence.1

Indisputable facts are difficult to ignore, indeed. Yet, facts are not always clear and unambiguous. Getting to the plain facts and drawing valid conclusions from them can be stubborn matters in their own right. To quote science teacher and YouTube lecturer, wonderingmind42, “Interpreting evidence well requires skill, training, and experience.”2 [Read more...]

—————————

1 In McCullough, D., 2001, John Adams, Simon & Schuster, p. 68. Red emphasis added.
2 Quote appears in the video at the 5:18 time mark. Also watch the segment from 2:40 to 4:20 about facts versus the interpretation of facts.

Discussing Accuracy

raylyons — Tue, 11 Jan 2011 22:07:29 +0000

Now that I am no longer distracted by the subject of last week’s entry I can get to the intended topic for my first 2011 blog entry. I should say, though, that I won’t be turning over any kind of new leaf for the new year. For now I’m sticking with the theme I’ve dwelt on already, a theme statistician Howard Wainer expressed concisely:

Whenever we discuss information we must also discuss its accuracy.1

Applied to the library world the message is: Formal library research and advocacy studies should always explain the strengths and limitations of their data. The best reason for abiding by this principle, in my opinion, is to allow readers to decide how much credence they want to give to conclusions drawn in the studies.

With new library advocacy studies on the horizon, I thought I better wrap up any unfinished topics from 2010. Howard Wainer’s advice made me think of the study, Opportunity for All: How the American Public Benefits from Internet Access at U.S. Libraries. Findings from this report have been broadcast widely with barely a mention of how approximate the figures are… [Read more]

—————————

1 Wainer, H., 2009, Picturing the Uncertain World: How to Understand, Communicate, and Control Uncertainty through Graphical Display, Princeton, NJ: Princeton University Press, p. 121.

Checking It Twice

raylyons — Wed, 22 Dec 2010 14:10:20 +0000

By this time in the season you’ve probably heard one or another rendition of the familiar Christmas carol about Santa’s annual performance measurement regimen. Mr. Claus and team work hard to make sure the results of the North Pole poll are accurate. After all, it would never do to have children receiving gifts they don’t want or deserve.

In his book, Proofiness: The Dark Arts of Mathematical Deception, Charles Seife points out that verification is also the cornerstone of good journalism. He writes, “Responsible reporters must take nothing for granted—only by basing every single sentence of your story upon observations or verifiable facts can you be assured that you’re reporting the truth.” Grittier newspaper veterans say it this way: “If your mother says she loves you, check it out.” [Read more]

A Hard Row to Hoe

raylyons — Fri, 05 Nov 2010 20:39:50 +0000

During the question-and-answer part of a presentation I gave at the 2010 Library Assessment Conference in Baltimore last week, I couldn’t resist editorializing about how bad convenience sampling is. One audience member spoke up, saying she felt convenience samples are legitimate as long as findings are interpreted as describing only the respondents, themselves. Later on I realized she was making a more interesting point, something that . . . [Read more]

Experience Keeps a Dear School

raylyons — Wed, 29 Sep 2010 03:09:19 +0000

Last June the final report from an IMLS-funded study of public library summer reading programs, The Dominican Study: Public Library Summer Programs Close the Reading Gap, was published. The “reading gap” refers to the cumulative loss in proficiency that has been observed in students who struggle with reading. The gap is cumulative because the “summer setbacks” that some students have add up, making them lag further and further behind good readers each year.

Researchers for the study, Susan Roman, Deborah Carran, and Carole Fiore, say that their main research question was, “Do public library summer reading programs impact student achievement?” The answer they delivered was, basically, “Yes.” Except the study does not actually demonstrate that the summer programs affect reading achievement at all… [Read more]

Stranger Than Fiction

raylyons — Wed, 17 Mar 2010 02:21:20 +0000

I proceed with this little essay with some trepidation due to the topic I’ve chosen: the idea of margins of error in survey research. By “survey research” I mean such things as political polls, public opinion surveys, market research, and so on.

Right up front I can share my conclusion with you. The common understanding of margins of error is incorrect. If you decide not to read this mini-exploration in its entirety, just remember this: Whatever you thought or heard that margins of error in surveys mean, simply abandon those ideas. Replace them with a mental question mark and you’ll be fine.

I say this because the precise meaning of margins of error is stranger than fiction. After all, the concept does come from statistical theory, which can definitely be on the fantastical side. Read on and you will see.