Just Make Up The Numbers

My prior post about the library research report, Opportunity for All, reminded me that as simple as percentages are, they sure can lead into bizarre territory. Suppose the report authors had determined that in 1990 an estimated 200 Internet terminals had been installed in U.S. public libraries nationwide. By 2007, then, the cumulative percentage growth in terminal installations would have been 103,900%!  Though this percentage would be perfectly factual, it is practically meaningless. Fortunately, the authors didn’t extrapolate back to 1990.

Percentages require us to pay attention to what baselines are used and how reasonable these are. Derrick Niederman and David Boyum emphasize this in their book, What the Numbers Say: A Field Guide to Mastering Our Numerical World, in the chapter, ‘Playing the Percentages.’  They also advise that venturing beyond percentages to percentiles requires extra vigilance:

Strange things happen with percentages of three digits or more, and they are often worthy of suspicion. Consider the ‘Mover and Shakers’ list on Amazon. com… Amazon decided that in addition to giving every book on their list a sales rank, they would alert people as to which books were showing the greatest upward movement in rank. One day in March 2001, the #1 Mover and Shaker was none other than Green Eggs and Ham, by Dr. Seuss. Alongside the book’s name was the numerical phrase ‘+4,310%.’

But what could that figure possibly mean? It turned out that…the book had gone from 441 to 10 on the list, and 441 is 4,310 percent larger than 10… In other words, in order to generate a big number, the conventional roles of [the fraction’s] numerator and denominator were interchanged…

As we emphasized in the previous chapter, most numbers count or measure something. Rankings, whether of sales, city populations, or figure skaters, are different. Like page numbers and the number at a deli showing which customer is being served, rankings belong to a special class of numbers called ordinal numbers. Ordinal numbers…simply indicate order and as such aren’t really numbers, in the sense that order can just as well be indicated by the letters of the alphabet. (As are the columns on a spreadsheet.) Now for the lesson: Be wary of doing arithmetic on ordinal numbers. Suppose that a book’s sales volume increased by 10 percent from one week to the next, moving it from 24th to 6th on a ranking list. To say the book advanced by 300 percent (the percentage by which 24 exceeds 6) is next to meaningless, precisely because the calculation is based on ordinal numbers, not the quantity of books sold…

A similar lesson applies to percentiles, which express rank in percentage terms. When a schoolgirl scores in the 90th percentile on a standardized test, it means she has done better than 90 percent of the kids taking the exam…It is tempting to assume that the 90th percentile is a much better performance than 45th percentile. But percentiles don’t tell us that. As a form of ordinal numbers, percentiles simply report rank and not underlying values. Depending on the circumstances, a 90th percentile performance could be dramatically superior to one in the 45th percentile, or there could be little difference between the two…In some cases, 90th percentile might signify an excellent outcome, while in other situations 90th percentile might be a disappointing result.

In ranking mutual funds, Morningstar awards a fund one to five stars based on a percentile ranking of the fund’s historical performance against similar funds… Because the star ratings1 represent percentile ranges, they don’t indicate the actual monetary performance of funds, either in absolute terms or relative to their peers.
In some fund categories…the difference in returns on a one-star and five-star fund might amount to no more than a percentage point a year over a five-year period. Within other categories…the gap in five-year returns between a one-star and five-star fund could easily exceed 20 percentage points annually. Moreover, a one-star fund in a strong category will often outperform a five-star fund in a weak category.

Neal Kaske and I described this exact problem in our article about public library ratings (see p. 40-41 in this issue of Public Libraries). Without the underlying data they represent, the informational value of rank-orders and percentiles is incredibly low. Adding, subtracting, multiplying, or dividing ranks or percentiles results in a kind of arithmetic gibberish.3 But this technicality doesn’t stop Amazon from calculating (literally stupid) percentages from ordinal ranks.

Think about it. If you placed 4th in a race and a friend placed 8th, would you brag that you were 200% faster than your friend? To the analysts at Amazon, though, an 8th place book moving upward four rungs to 4th place is a better seller by 200%. Yet a book moving from 20th to 16th place advances by only 25%, while one moving from 500th to 496th place advances only a miniscule 0.8%. Unevenly spaced rungs on this measurement “ladder” trip us up. In the meantime, the actual sales increases are obscured by these squishy percentages.

The time is long past when we are surprised by corporations using numbers to pull the proverbial wool over our eyes. Still, companies like Amazon and Morningstar are especially brazen because they claim to provide consumers with information to improve buying decisions. The success of tricks and junk calculations like theirs depends on the gullibility of their audience. Until information consumers become quantitatively literate, corporations and institutions will be happy to just make up the numbers and see what they can talk people into believing. This Dilbert cartoon shows how easy this is.


1  The LJ Index of Public Library Service avoids percentiles completely. LJ Index scores reflect the underlying data directly as I explained here. Please don’t confuse Morningstar’s stars with Library Journal’s. As constellations go, the LJ Index is a Pegasus of a different color!
2  Niederman and Boyum, 2003, p. 93-97. Blue emphasis added.
3  To non-mathematicians, I mean to say. Ordinal numbers can indeed be added, subtracted, multiplied, squared, etc., but the results are not everyday arithmetic at all, as you can see here.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s