How to misuse 311 data

The Bed-Bug (Cimex lectularius)

I read a recent article in the Chicago Reader about continuing bed-bug infestations that inspired me to comment on an easy-to-make mistake regarding 311 data. Here’s what the article says:

The City of Chicago’s Department of Buildings tracks the number of bedbug infestations reported through 311 calls, and reports [an upward trend].

The department started keeping a record in 2006; there were 25 calls that year, 50 the next, and 103 in 2008. Since then the number of calls has increased by roughly 100 each year, totaling 376 in 2011.

(This information was further highlighted in an infographic embedded in the article.)

Here’s the problem: The author seems to suggest that the increase in 311-bed-bug reports is evidence of an increasing bed-bug problem, but the number of 311 calls per year fluctuates. So it’s impossible to know whether there are more bed bugs or whether simply, for some other reason, more people thought to call 311 in a given year. Perhaps a local tv station publicized 311 that year, thus driving up calls. I was unable to find reliable data on the total number of 311 calls for 2006 to 2011, but I know the numbers for 2008 (4,533,125) and 2009 (4,136,505), showing that the yearly call volume can vary by nearly $400,000 year-to-year.

The better metric would be the increase in the ratio of bed-bugs reports to total 311 calls. At least that would account for the possibility that people were just using 311 more in general during a certain year. Based on my research, I still think there is an upward trend, though maybe not for 2010 to 2011, when the increase in bed-bug calls was only 76 calls.

One resource I find particularly helpful on matters like this is Darrell Hunt’s classic “How to Lie with Statistics,” which teaches the reader, in a fun and readable way, to be skeptical of how of the media presents data. It should be required college reading.

5 Replies to “How to misuse 311 data”

  1. From “Matt L.” @ old blog:

    I’m not sure that looking at the absolute number of bed bug calls isn’t the best way to look at this. You’re theorizing that the year to year call fluctuation could have a proportional effect on bed bug calls. The fluctuation from ’08 to ’09 was less than 10%, yet the increase in bed bug calls is much larger than 10%. Also one cannot assume that if calls go up 50% there will be any effect on total bed bug calls. There could be another problem (a rabies outbreak for instance) that drives up call volume in a particular year.
    I agree that media attention toward this particular issue will most likely have an effect on number of calls (especially if they tell people to call 311), but there are plenty of other sources cited in the article that point to a legitimate increase.

  2. My response: I agree with you that the ’08 to ’09 data still shows an upward trend and that the article points to plenty of other sources; I’m not saying there isn’t an increasing bed-bug problem. The problem I see is highlighted by the infographic called “The Ballooning 311 Complaints about Bed Bugs,” which shows the increase from ’10 to ’11 was only 76 calls. 76 calls out of 4 million is minuscule; it really shows very little. The infographic was much larger in the print edition of the paper (where I read the article first), clearly suggesting this “ballooning” showed an increase in bed bugs. I’m simply saying that a 76-call increase from ’10 to ’11—without knowing whether 311 calls rose proportionally in ’11—leaves way to many open variables to show a meaningful correlation.

  3. Benton, I take your point that the volume of bedbug calls to 311 could fluctuate year by year with the total volume of 311 calls, but the number has been steadily increasing year over year since 2006. The increase in bedbug calls reported by the city’s building’s department is also consistent with the increases that the Metropolitan Tenants Organization and the Illinois Department of Health have seen. In fact, we were originally planning to do a graph with numbers from all three, and they were so similar that it didn’t seem worth it to include them all in the graphic. I ended up reporting the numbers from MTO and the Illinois health dept in the text only.

  4. The data reported in the article is incorrect about the number of calls. The number of calls in the first three quarter for 2011 was 1022 . CDPH tracks bed bug calls separately and they have graphs available for most of 2010, and all of 2011. If you give me your email I can send you the actual maps. In addition CHA does not report through 311 and those numbers are therefore not tracked thus reducing that amount by hundreds of complaints.

Leave a Reply

Your email address will not be published. Required fields are marked *