Medical Trial and Error

The Atlantic has an interesting profile of medical researcher John Ioannidis, who famously concluded in a groundbreaking 2005 study (a different version of which can be found here) that the scientific conclusions of most of the articles published in medical journals are wrong.  I’m not sure what took The Atlantic so long to write the review.  After all, The New York Times Magazine covered the same ground way back in 2006, and the Times excerpted a lengthy section of Ioannidis’s new book on the subject in June of this year.  Still, the conclusions are worth taking into consideration.  After all, science is merely an organized search for the truth, and every scientist worth his salt knows that any one study can generate incorrect or incomplete results.  It is only once a given observation is tested repeatedly and reproduced that we should come to view the findings as anything more than provisional.

What’s troubling is the reception that Ioannadis’s work has gotten from quacks and cranks — exemplified by a handful of the comments following The Atlantic article, and elsewhere.  If science can’t be trusted, then science must be useless, according to some.  Others lay the blame at the feet of pharmaceutical companies peddling junk science in order to sell more drugs.  But, as pharma researcher and blogger Derek Lowe points out, “drug research probably comes out of [Ioannadis’s] analysis looking as good as anything can. A large confirmatory Phase III study is, as you’d hope, the sort of thing most likely to be correct, even given the financial considerations involved. Even then, though, you can’t be completely sure – but contrast that with a lot of the headline-grabbing studies in nutrition or genomics, whose results are actually more likely to be false than true.”

In the end, the cumulative weight of a body of science, as opposed to any single study, still is right more often than not.  To quote science blogger Orac, “in many ways, the present system of randomized clinical trials and peer-review is, to paraphrase Winston Churchill regarding democracy, the worst system for finding the best treatments–except for all the rest. … Despite all that, it is a big mistake to take Ioannidis’ findings as “proof” that science is not the best methodology we have for answering fundamental questions about how the universe works, the pathogenesis of disease, or for identifying the most efficacious treatments. Certainly, it far surpasses any alternatives.”