Junk Science: Military Suicide Mystery

Researchers and the media did their best this week to scare military personnel and their families with the widely reported headline, “Military Service Doubles Suicide Risk.”

“Male veterans are twice as likely as their civilian counterparts to die by suicide,” Portland State University professor Mark Kaplan told the Atlanta Journal Constitution. “We don’t know why. But this finding may foreshadow what is going to come with the current cohort of military personnel who have served in Afghanistan and Iraq,” he added.

Published in the Journal of Epidemiology and Community Health (July 2007), Kaplan’s study consisted of 320,890 men who were followed for 12 years. As it is a statistical correlation study – rather than an investigation into whether an actual cause-and-effect relationship exists between military service and suicide – I naturally was skeptical. From the very beginning, the study didn’t disappoint me.

The study summary stated that the veterans’ suicide rate was 2.04 times that of non-veterans. When I read the study to see how the 2.04 figure was derived, I found no explanation. Mysteriously, the 2.04 figure did not even appear in the study itself – that’s pretty unusual.

I did, however, find a bar graph in the study that presented 2.13 as the difference in suicide rate between veterans and non-veterans.

You might think that this solved the mystery. A typographical or editorial foul-up must have inadvertently led to the 2.04-figure, rather than the 2.13 figure, being spotlighted in the study summary, right? We’ll get to that later. In the meantime, my discovery of the 2.13-figure only deepened the mystery.

Kaplan wrote in his study that the 2.13-figure represented the difference in suicide rates between veterans and non-veterans after statistical adjustment to account for other potential risk factors for suicide, including age, marital status, living arrangement, race, education, family income, employment status, geographic region, interval since last visit to a doctor, self-rated health and body mass index.

This list seemed impressively comprehensive and ostensibly strengthened the case for his claimed result – until, that is, I discovered that a key potential suicide risk factor apparently was omitted from his statistical adjustment.

There’s a table in Kaplan’s study in which he presents the difference in veteran suicide rates by individual risk factors, including age, race, marital status, living arrangement, education, employment status, region of residence, urban/rural locality, self-rated health, body mass index, psychiatric conditions and activity limitation.

With the exception of race, education and activity limitation, none of these risk factors were statistically significantly associated with increased suicide rates. But since race, education and activity limitation were associated with increased suicide risk, all three should have been among the potential risk factors Kaplan considered when he did his statistical adjustments to produce the 2.13-figure.

If you compare the above-mentioned lists of suicide risk factors, however, you’ll note that while activity limitation was identified as a significant risk factor for suicide, it apparently was not included in the statistical adjustment that produced the 2.13-figure.

And of the three statistically significant risk factors for suicide, activity limitation was by far the greatest – veterans with activity limitations had a 4.44 times greater rate of suicide than veterans with no activity limitations, as compared to race (3.23) and education (2.67).

Is the omission of the activity limitation factor another study typo? Was it inadvertently omitted from the statistical adjustment? Or was it omitted from the analysis because it would produce a non-result that rendered the study non-publishable and non-newsworthy?

It certainly cannot be said that Kaplan was ignorant of the significance of the activity limitation risk factor. “According to Kaplan, the risk of suicide was highest among men whose activities were limited by health problems,” reported the Atlanta Journal-Constitution.

Kaplan also published a study earlier this year entitled, “Physical illness, functional limitations and suicide risk: A population-based study” in the American Journal of Orthopsychiatry (Jan. 2007) in which he stated, “After controlling for potential [confounding risk factors], functional limitations were shown to be a significant predictor of suicide.”

When I contacted Kaplan about these issues, he immediately acknowledged that the 2.04-figure was a typo and that the 2.13-figure was correct. Interestingly, he also provided me with a dubious error bar for the 2.13 figure. When I asked him about that, another acknowledgment of error was made. These may seem like small errors, but they certainly build no confidence.

As to the crucial omission of activity limitation as a risk factor, Kaplan deferred responding, writing that he needed to consult with one of his statistician co-authors.

As of the time of this column, I had not heard back from Kaplan on that point. But you might think that a lead study author who gave many media interviews this week would be readily familiar with such a key component of his analysis. Of all the researchers I’ve interviewed over the years about their results, none has ever failed to immediately provide an answer to such a basic question.

I don’t know whether Kaplan ultimately will produce a satisfactory explanation for the activity limitation omission – the study’s remaining mystery. In some ways it doesn’t matter.

The study’s other shortcomings – particularly that veteran suicide rates weren’t higher across the vast majority of demographic groups examined, which indicates that military service itself isn’t a causative factor in suicide – are alone enough to debunk it and the scary headlines it spawned.

But the wide reporting of a paper with such major and easily discoverable problems – as well as Kaplan’s questionable effort to foment concern about suicide risk among veterans of Afghanistan and Iraq – reflects poorly on him and his co-authors, the publishing journal and the media.