I read a recent article in the Chicago Reader about continuing bed-bug infestations that inspired me to comment on an easy-to-make mistake regarding 311 data. Here’s what the article says:
The City of Chicago’s Department of Buildings tracks the number of bedbug infestations reported through 311 calls, and reports [an upward trend].
The department started keeping a record in 2006; there were 25 calls that year, 50 the next, and 103 in 2008. Since then the number of calls has increased by roughly 100 each year, totaling 376 in 2011.
(This information was further highlighted in an infographic embedded in the article.)
Here’s the problem: The author seems to suggest that the increase in 311-bed-bug reports is evidence of an increasing bed-bug problem, but the number of 311 calls per year fluctuates. So it’s impossible to know whether there are more bed bugs or whether simply, for some other reason, more people thought to call 311 in a given year. Perhaps a local tv station publicized 311 that year, thus driving up calls. I was unable to find reliable data on the total number of 311 calls for 2006 to 2011, but I know the numbers for 2008 (4,533,125) and 2009 (4,136,505), showing that the yearly call volume can vary by nearly $400,000 year-to-year.
The better metric would be the increase in the ratio of bed-bugs reports to total 311 calls. At least that would account for the possibility that people were just using 311 more in general during a certain year. Based on my research, I still think there is an upward trend, though maybe not for 2010 to 2011, when the increase in bed-bug calls was only 76 calls.
One resource I find particularly helpful on matters like this is Darrell Hunt’s classic “How to Lie with Statistics,” which teaches the reader, in a fun and readable way, to be skeptical of how of the media presents data. It should be required college reading.