Last month, researchers at the University of Florida published a study in the American Journal of Public Health that concluded, “Increases in alcohol excise taxes, such as the 2009 Illinois act, could save thousands of lives yearly across the United States as part of a comprehensive strategy to reduce alcohol-impaired driving.” Their study presented the case that the 2009 tax increase resulted in a statistically significant reduction in alcohol-related deaths in Illinois. However, as I pointed out in a blog post, there only appears to be a reduction in fatalities because of the authors’ selective inclusion and exclusion of data. Rebecca Goldin, Director of STATS.org and Professor of Mathematical Sciences at George Mason University, found even more flaws in the research.
In her post, Rebecca takes a bird’s eye view of the crash data in Illinois between 1999 and 2013. She finds a steady decline during this 10-year period with “hardly anything special at all going on at the end of 2009.” Looking closer at the data, she noted (as did I) the curious exclusion of data after 2011. “Results should be resilient to changes in choices, such as whether to use data published 2011-2013,” Goldin notes. When added to the evaluation, this 2011-2013 data shows a small decrease immediately after 2009, but then shows a steady increase in alcohol-related traffic deaths in Illinois. She also found that as a proportion of total traffic fatalities, alcohol-related fatalities on the road have been increasing since 2009 (though only by 2 percent—a statistically insignificant amount).
Of course, the real story here is that breaking the data at year 2009 may not be an appropriate way of measuring the trends, and certainly it’s not appropriate to “blame” the excise tax for the proportionally increased alcohol deaths in the context of decreasing deaths associated to alcohol. But by the same logic neither is it appropriate to conclude that the excise tax has been saving lives.
Finally, in addition to the “creative math” and error of conflating correlation with causation (bad scientists, bad!), there’s another potential problem with the study. The grant provided by the National Institutes of Health lists the purpose as “to develop a research program of HIV prevention focused on the intersection of event-level alcohol use… and sexual partner selection among adolescents.” There is no mention in the grant description about taxes or motor vehicle crashes. I say this is a “potential” problem because I have been unable to get answers from anyone at the NIH or the University of Florida researchers who worked on the project. My questions were simple: What was the rationale behind using an AIDS grant for alcohol tax research? And is this sort of change in scope common practice?
It’s one thing to have public health advocates peddling flawed research in order to advance personal agendas, but it’s another entirely to use taxpayer money to do so.