Recent months have seen some regrettable lapses by prestigious scientific journals. Some highly questionable claims have been made, but have been published anyway. These suggest that the time-honored system of journal publication is breaking down in the face of two distinct but complementary pressures. This in turn means that science is running the risk of losing its value to civilization.<?xml:namespace prefix = u1 /> <?xml:namespace prefix = o ns = “urn:schemas-microsoft-com:office:office” />
Two examples of the lapses (there are many more) serve to illustrate the point. First is the retraction of the widely-publicized claim by the Centers for Disease Control and Prevention (CDC) that obesity was beginning to rival smoking as a cause of early mortality in the <?xml:namespace prefix = st1 ns = “urn:schemas-microsoft-com:office:smarttags” />US. The headline figure of 400,000 deaths each year was cited as evidence that Americans “were eating themselves to death.” It became a shibboleth by which diverse groups of lifestyle fanatics recognized each other. Yet there was one group of health campaigners that sniffed something wrong, the anti-tobacco groups.
According to The New York Times, those groups thought the obesity numbers had been seriously fattened up, with one source putting the true number at 100,000. It appears that the CDC has accepted their criticism, and the figure will be revised downwards. Yet the error should have been caught earlier. There were several serious problems with the CDC study, which appeared in the Journal of the American Medical Association in March 2004. It was based on a roundly-criticized study from 1999. Its data went back to 1948 and made no effort to correct for medical advances since then. The study did not control for obvious confounding factors and it also included 60,000 deaths of people classed as overweight, something that has no statistically significant correlation with early mortality.
The study itself, to be fair, was not just about obesity (it was about all causes of early death), but it should have been obvious to reviewers and editors that something was up with its methodology when it found obesity was so close to overtaking tobacco as a leading cause of early death. As the Times points out, the study used different methods to calculate the mortality figures for tobacco and diet. This should in itself have set alarm bells ringing. The fact that it was published raises serious questions about JAMA's peer review process.
The second was the Lancet's early publication, on the eve of the US Presidential election, of a study that found large increases in the numbers dying prematurely in Iraq after the removal of Saddam Hussein. The study itself is actually much more statistically sound than many commentators (including some in these pages) have suggested, and it certainly suggests that the mortality rate is worse in the unstable insurgency-ridden Iraq after the ouster of Saddam's regime than during the last days of his tyranny.
Yet this is, in a way, not surprising. Conquests are normally followed by messy aftermaths, which exacerbate conditions that may have been kept under control by even the worst of regimes. Saddam's tyranny was for the most part about control via torture and prisons, not death camps (although it had those, too). It is particularly galling that one criticism that has been advanced is the comparison of the pre-war child mortality figures cited by the study with the debunked UNICEF figures from 1999, which were regularly advanced by Saddam apologists as reasons for the lifting of sanctions (see Matt Welch's excellent article on the subject). That many of the deaths are attributable to coalition military action is also unsurprising. Large areas of the country are still the subject of armed struggles. As General Sherman reminded us, war is all hell.
The problem was not so much with the study itself as with its presentation by the journal. The study extrapolated a wide range of possible excess deaths, between 8,000 and 194,000, with a most-likely figure of 98,000. The Lancet, however, touted the study as finding 100,000 “civilian deaths.” This is wrong from two standpoints. First, the study is about deaths, pure and simple, and did not distinguish between combatants and non-combatants, so the editors were inserting an interpretation that could not be supported by the study. Second, the use of a rounded-up attention-grabbing simplification of a complex issue points clearly towards the study being used for non-scientific purposes. Lancet editor Richard Horton argued in his commentary that the study “requires an urgent political and military response.” The timing of the study's online publication, however, suggests that this was a base attempt by Horton and his colleagues to influence another country's election (The Lancet and most of its staff are British) just as ham-fisted as the Guardian's Operation Clark County.
As I said, there appear to be two distinct but complementary trends here. The first has always happened in science. The late Thomas Gold referred to the “herd instincts” of scientists, while Thomas Kuhn, more formally, argued that science progresses by dramatic paradigm shifts, where new theories initially meet resistance but then become accepted (which means that the vast majority of scientists are always on one side of an issue, right or wrong). This means, however, that science that is acceptable to the scientific herd is less likely to be criticized than that which challenges the herd's preconceptions. This is normally a good thing, for obvious reasons.
However, there is also a trend to make science about more than data or objective testing. The journal Nature recently endorsed a suggestion by left-wing British think tank Demos that scientific research priority-setting should involve significant input from the general public. This is the very definition of the politicization of science. At the same time, scientists seem increasingly likely to engage in political discourse, demanding that science shows that certain political actions must be taken. This increasing politicization of science, combined with what has been called the “scientization” of certain issues, joins with the herd instinct to distort science.
Thus, because fast food is seen as a problem by anti-globalization campaigners, science that confirms obesity as a serious public health problem becomes less likely to be examined as critically as it should be. The foreseeable consequence is that when it is revealed that the problem has been overstated the reputation of science suffers. Similarly, perfectly good science like the Iraq study has been the subject of unfounded criticism and dismissal primarily because of the political uses to which it was put. Using science to advance popular agendas diminishes the value of the science.
The journals play a central role in exacerbating these problems. Their peer review procedures are failing too often, mainly, it seems, in politically sensitive areas. At the same time, their mishandling of studies for political purposes threatens to devalue their contribution to the sum of human knowledge.
As MIT scientist Richard Lindzen has said, “Science is a tool of some value. It provides our only way of separating what is true from what is asserted. If we abuse that tool, it will not be available when it is needed.” It is unfortunate that the journals currently seem to be engaging in serial abuse of science.