Opinion research guru Bob Lichter has a good dissection of the problem with the Union of Concerned Scientists study on federal interference with science that made headlines last week here. Of particular note are his comments about why even the "absolute" numbers of incidents the study uncovered are suspect:
There is also a serious problem with the questionnaire itself. Scientists were asked not only about their own experiences but about their perception of others. For example, the Los Angeles Times reported that "nearly half of all respondents perceived or personally experienced pressure to eliminate" certain words. The actual wording was, "I have perceived and others and/or personally experienced the following types of activities affecting climate science:" This was followed by a list of activities that the scientists marked as "perceived" "experience," or " neither." This wording creates a statistical phenomenon that artificially inflates the impression of a hostile work environment. Consider an agency that contains 10 scientists. One tells the other nine that he has encountered interference. When they are surveyed, all ten report that they have "perceived in others and/or personally experienced" interference. So one act of interference is counted as ten acts that are "perceived" or experienced"; ten percent of the scientists have been interfered with, but 100 percent report "perceiving in others and/or personally" experiencing interference. If the agency contains 100 scientists, the interference experienced by one becomes "perceived" interference by the other 99, and so forth. Further, there is a great deal of social scientific evidence that such perceptions are often incorrect. The tendency to believe that others will be influenced by forces to which we ourselves are immune (e.g., by misleading advertising or partisan rhetoric) is so common that sociologists have a name for it -- the "third person effect." In addition, people's perceptions of their own situations may be colored by their prior beliefs and emotions. In fact, out of the 12 "activities" listed, nearly half are actually mental states or inferences: "self-induced pressure to change research," "fear of retaliation" for expressing concerns either inside or outside one's agency (listed as two separate activities), "implicit expectation by officials for scientists to provide" misleading information. (Of course, the researchers also assume that the scientists were always right and their superiors always wrong in urging them to eliminate certain words, requesting that they insert opposing views, making changes and edits etc.) Finally, the researchers assume that all these responses refer to officials' efforts to alter certain kinds of findings about global warming. But that is not specified in the questionnaire. For example, a scientist could encounter "new or unusual administrative requirements or procedures that impair climate related work," without the requirements having anything to do with the debate over global warming.Lichter also does a good job in explaining exactly how useless the mainstream media were in covering this study.