CEI Comments on EPA’s Supplemental Notice of Proposed Rulemaking : EPA Transparency Rule

View Full Document as PDF

Thank you for the opportunity to comment on the Environmental Protection Agency’s (EPA) Supplemental notice of proposed rulemaking (SNPR)[1] on its proposal to strengthen transparency in regulatory science.[2] The comments below are intended to help EPA rebut spurious criticism of its proposal. To that end, I will quote or summarize, then respond to, claims made by opponents of the Transparency proposal.*

Senate Environment and Public Works Committee Hearing

The Senate Environment and Public Works Committee held a hearing on the proposed Transparency Rule on October 3, 2018—roughly six weeks after the comment period closed.[3] 
The rule’s basic objective is to ensure that, over time, more of the data and models underlying EPA’s regulatory science are available for independent validation in a manner that is consistent with law, patient privacy, and confidentiality agreements between researchers and subjects (83 FR 18770). An unwitting piece of evidence in support of the proposal was the non-stop effort by Committee Democrats and the minority witness to defame the rule. 
 
According to Ranking Minority Member Sen. Cory Booker (D-NJ), the transparency rule would bar EPA from using scientific studies “unless the underlying data is [sic] made publicly available in a way that is sufficient for validation.” Consequently, “the agency would not be able to consider science gathered in the aftermath of environmental disasters, such as the Deep Water Horizon oil spill, which is not a scientifically-replicable event.” Nor would the agency “be able to consider studies that rely on private medical information or confidential business information because that data cannot be made publicly available.”
 
Indeed, Sen. Booker suggested, the rule might prohibit EPA from regulating unless the agency first conducts unethical experiments: “And obviously, it would be unethical for anyone to attempt to replicate public health analyses that use data gathered from different exposures to certain populations and communities—exposures to lead, to PCBs, to mercury or other chemical contaminants. We would not want anybody to replicate those studies, and that suffering.” 
 
In any event, Booker continued, the transparency rule could “cripple” EPA’s efforts to protect children from environmental hazards: “For example, EPA bases its standards for lead-based paint hazards on long-term studies of children who are exposed to lead. Prohibiting EPA from using these historical studies would cripple its ability to protect children and other populations from lead, as one example.”
 
Such criticisms are preposterous. All the rule attempts to do is increase the opportunity for independent researchers to examine such data, not to replicate the data by causing similar disasters, or subjecting children to similar hazards.
 
The minority witness, Dr. Rush Holt, CEO of the American Association for the Advancement of Science, claimed “the effect of the rule would be a reduction in good relevant science that could be used by EPA.” That, in turn, “would likely result in harm to people and the environment.” How so? Supposedly, because “The proposed rule in its strict application would allow only research that is made completely public. And this demonstrates either a deep misunderstanding of how science works and should work. Or an intention to cherry pick evidence in the name of transparency.”
 
The root of the problem, Holt suggested, is that the transparency proposal confuses “verification” with “replication.” Science, he explained, “has to be empirical, based on experiment, observation, and then it has to be verified. That’s the key word. It’s really a red herring to say replication is what is necessary. The verification can come in various ways. Through repeating the experiment if it’s an experiment. But even most experiments are hard to repeat exactly. Certainly, natural disasters—Senator Booker referred to the Gulf oil discharge—let’s hope that isn’t repeatable.”
 
That criticism, too, is ridiculous, as is easily demonstrated because the Transparency proposal is only seven pages long. Besides, majority witness Robert Hahn of the Brookings Institution had rebutted those allegations nearly five months earlier in the Washington Post: “Here’s what the EPA’s rule wouldn’t do: nullify existing environmental regulations, disregard existing research, violate confidentiality protections, jeopardize privacy or undermine the peer-review process.”[4]
 
Nowhere in the proposal does EPA state or imply that non-replicable studies should not be considered in rulemaking. EPA knows that under the Administrative Procedure Act (APA), it must consider all comments submitted on the record, the vast majority of which do not purport to replicate data, and nearly all of which are not peer-reviewed.

Nowhere does the Transparency proposal say or imply that all aspects of a study must be “completely public” to be used. The key concept in the proposal is independent validation. In the Supplementary notice, EPA clarifies how a system of “tiered access” would allow qualified independent researchers to review personally identifiable information (PII) under the same confidentiality requirements and legal sanctions that prevent the original researchers from compromising medical privacy (PII) (85 FR 15399).

Dr. Holt, who may have known he was attacking a strawman, tried to finesse matters by claiming the proposal in “its strict application” would exclude research that is not “completely public.” No. Strictly applying the proposal means applying it as written. It does not mean applying a caricature.
 
“Peer review” is one of the methods Dr. Holt considers sufficient for “verification.” However, when the peers are colleagues who co-author each other’s work, share the same methodological or policy commitments, and depend on the same funding sources, peer review may be little more than pal review.
 
Accordingly, to the full extent practicable and allowed by law, the pivotal studies underpinning significant regulation should also allow for post-publication audit by independent researchers. As in the marketplace of goods, so in the marketplace of ideas, competition promotes quality. Rival researchers are more likely than journal editors and agency peer reviewers to discover flaws in a study’s data or models.
 
Because every participant in the regulatory process is a stakeholder—an interested party who either is or has a dog in the fight—“trust us, we’re the experts” is no substitute for competitive and independent vetting of data and models. The transparency proposal thus solicits comment on “how to incorporate stronger data and model access requirements into the terms and conditions of cooperative agreements and grants.” But according to Dr. Holt and Sen. Booker, the Transparency rule will exclude EPA’s access to legitimate science and children will die.
 
Such melodramatic attacks on a common-sense proposal only serve to underscore the Transparency Rule’s desirability. That the self-appointed defenders of science speak with such reckless disregard for the facts is all-the-more reason why strong safeguards are needed to ensure independent validation of regulatory science.

Union of Concerned Scientists Zoom Teach In

E&E News on April 15 reported the main takeaways from an all-day Zoom hearing on the Transparency proposal hosted by the Union of Concerned Scientists.[5] The drum beat message of the 40 participating experts was that the Transparency Rule would create a public health crisis.

For example, as summarized by E&E, Stanford Children’s Health doctor, Lisa Patel, warned that the Rule would “bar the agency from using the latest scientific research on air pollution and disinfectants amid rapidly evolving public health crises.” In other words, the Transparency Rule would somehow prohibit EPA from posting and updating a list of recommended disinfectants for use against COVID-19.[6] That is beyond ridiculous. The Transparency Rule applies only to “pivotal regulatory science” and “influential scientific information” (85 FR 15405). Nothing in either the original proposal or the subsequent supplemental would prevent EPA from recommending the use of hydrogen peroxide or ammonium chloride (the chemical in disinfectant wipes) to kill viral pathogens on contaminated surfaces.

Let’s examine some additional claims from the UCS briefing. Excerpts from the E&E article are flush left in italics. Misleading verbiage is underscored. Each of my responses is labeled “comment” and indented.

Forty public health advocates, environmentalists and doctors tuned in to the hearing to condemn the proposal, which would limit EPA from crafting environmental or public health protections with scientific research that has not been made public.

Comment: If the “experts” said that, they are attacking a straw man. Making data and models available to qualified researchers for the purpose of independent validation is not the same as posting confidential business information or personal health data on public Web sites.

The rule has become one of the most controversial efforts at the agency because it could fundamentally influence rulemakings for years to come. The concept has long been pushed by Republicans, and critics charge that the tobacco industry first pedaled the idea in an attempt to suppress data that hurt its brand. 

Comment: If the critics said that, they peddle ad hominem, guilt-by-association innuendo in the name of science. It matters not whether tobacco lobbyists used the word “transparency.” They probably also said the Sun rises in the East. From time immemorial, people in adversarial proceedings have said, in one form or another, “Show me your data!” Such skepticism is essential to both science and self-government.

A more relevant historical precedent is the Data Access Act, a provision of the fiscal year 1999 appropriation for the Office of Management and Budget (OMB). The provision required OMB to amend Circular A-110 to require all federal awarding agencies “to ensure that all data produced under an award will be made available to the public through the procedures established under the Freedom of Information Act.”

The basic rationale, as summarized by OMB in its implementing rule, is that “the public has a right to obtain research data that have been funded with tax dollars, particularly when the research findings were used by the Federal Government in developing policy or rules.”[7] Circular A-110 exempts confidential and personally identifiable information from FOIA.[8]

Nonetheless, the usual suspects raised the same objections now being leveled at the Transparency proposal. They claimed OMB’s revisions “would harm the traditional process of scientific research; human subjects would believe that the federal government might obtain access to confidential information; researchers would have to spend additional time and money putting data into a form required by the government, thereby interfering with ongoing research; and private sector cooperation and funding for government/university/industry partnerships would be jeopardized.”[9] None of those dire prognostications came to pass.

Two years ago, former EPA Administrator Scott Pruitt unveiled his plan to eliminate “secret science” from regulations. The current head of EPA, Andrew Wheeler, has since fleshed it out and expanded its scope. Proponents have continued to call it the “secret science” proposal, while critics have largely adopted the moniker “censored science.”

Comment: Perhaps a better way to describe what EPA seeks to discourage and its critics seek to protect is “trust me science.”[10] That moniker points to the elitism of those who refuse to let the great unwashed (i.e. independent researchers) peek behind the curtain (examine their data).

Yesterday, at the virtual hearing, Rep. Paul Tonko (D-N.Y.) called [EPA’s Transparency Rule] “selective science,” suggesting that the rule would allow the agency to pick and choose which research is valued at EPA.

Comment: Agencies already pick and choose which studies are valued. For example, agencies typically prefer peer-reviewed studies to non-peer-reviewed studies, a practice no one objects to. Sometimes, however, agencies prefer agency-led and -funded studies, which is problematic. It is within EPA’s reasonable discretion to decide that, other things being equal, studies accessible for independent validation are more valuable than trust-me-science studies. Although the APA requires EPA to consider all significant comments, nothing in the APA prohibits EPA from establishing quality control standards that give preference to studies that, in addition to being peer-reviewed, have also been vetted by independent researchers.

Other critics raised concerns about the long-term implications for science. Paul Billings of the American Lung Association warned that the rule could deter people from participating in important studies out of fear that they could be identified. For example, the National Institutes of Health is currently looking for 10,000 Americans to see if they might have contracted the coronavirus without knowing it. “It may stifle or reduce participation in public health studies,” he said. “It may have a chilling effect on research.”

Comment: That makes little sense. Why are people now willing to share personal health data with epidemiologists? Chiefly because the researchers are bound by legally enforceable confidentiality agreements. There is absolutely no reason why auditors of such studies would not be similarly bound and liable for breaches of confidentiality.

The issue of privacy of human subjects has been central in critics’ complaints. Wheeler’s latest draft sought to assuage those concerns by proposing a tiered-access approach to handle confidentiality. But critics remain unconvinced. Patrice McDermott of the Government Accountability Project talked about the difficulty in de-identifying data in many cases. And Roy Gamse, a former deputy assistant administrator of EPA, said research has shown how easy it is to identify study participants. “People don’t know if the data will be anonymous if a malicious hacker tries to hack it. So, it cannot guarantee confidentiality,” he said. “It can only promise best efforts.”

Comment: If hacking is the problem, that applies to original investigators no less than to those who audit published studies. Modern epidemiologists do not type their research on IBM Selectrics, store data solely on hard copy in filing cabinets, or refuse to share data electronically with co-authors. Thus, researchers who oppose the Transparency Rule cannot guarantee confidentiality either. All they can offer study participants is the promise of best efforts. The critics fall to their own critique.

Deborah Wallace, who has a Ph.D. in ecology, added that the proposal would thwart meta-analysis, or research that analyzes studies across disciplines, because many studies would be excluded.

Comment: A meta-analysis is the use of statistics to summarize the results of other studies. Nothing in the Transparency proposal would prevent EPA analysts from conducting or reviewing meta-analyses. The new emphasis on independent validation might, however, make meta-analyses more useful by making the underlying studies more reliable.

“The rule opens standards and processes to reanalysis and alternative models, including by well-funded consultants and regulated industries,” she added, but noted that it does not provide assurance for conflicts of interest.

Comment: Conflicts of interest in the rulemaking process are ubiquitous. Although participants may be honest people, few are honest brokers—persons without an interest or stake in the outcome. Stakeholders include not only the industries subject to the agency’s regulations but also the advocacy groups that gain influence and contributions by supporting or opposing the agency’s agenda, the researchers who receive agency grants to study issues of regulatory interest, and the agency itself, whose scientific assessments drive the scale and scope of its regulatory activity.

Although every stakeholder is an interested party, disclosure requirements that bring conflicts of interest to light can help make the process more competitive. Nothing in the Transparency Rule would change any conflict of interest disclosure regulations or any part of EPA’s scientific integrity policy.[11]

The hearing was held the same day EPA announced it would not update regulations for fine particulate matter, or PM2.5 (Greenwire, April 14). Patel, the doctor, stressed that the best available science shows us that microscopic air pollutants are more dangerous than previously known. And new research shows that air pollution can cause up to 15 percent increase in COVID-19 mortality rates. “The EPA rule would essentially quash this type of research if there was no hope for it to be utilized to change policy,” she said.

Comment: Dr. Patel refers to a Harvard study, “Exposures to air pollution and COVID-19 mortality in the United States,” released on April 5.[12] The authors state on page 5: “All data sources used in these analysis, along with fully reproducible code, are publicly available to facilitate continued investigation of these relationships as the COVID-19 outbreak evolves and more data become available.” Consequently, the Transparency Rule would not “quash” the study because the latter already fully complies with the Rule’s letter and spirit.

Within weeks the study’s release, the researchers revised their mortality estimate. As reported in the Washington Post, “Instead of linking an increase in exposure of one microgram per cubic meter to a 15 percent greater likelihood of dying of covid-19, the team said instead it is associated with an 8 percent increase in mortality.”[13] Does anyone believe the researchers would have revised their results as quickly if independent researchers had been barred from examining the study’s modeling and data?

Democrats and public health advocates touted the study upon its release, but they also applauded a March 2020 study claiming the coronavirus lockdown saved 77,000 lives in China just by reducing PM2.5 pollution.[14] The consistency between the two studies is not obvious. Long-term average PM2.5 levels in the United States have been at or below 12 micrograms per cubic meter (µg/c3) since 2012.[15]  

Annual PM2.5 levels in Wuhan averaged 89.6 µg/c3 in 2017.[16] So, if an additional one microgram per cubic meter increase in long-term PM2.5 exposure increases coronavirus deaths by 8 percent, Wuhan’s PM2.5 levels should have produced 7 times the number of U.S. coronavirus deaths per unit of population. Yet the March 2020 study concluded that the drop in short-term PM2.5 levels from the Wuhan lockdown avoided 77,000 premature deaths.

Skeptical observers may be forgiven for suspecting that both studies seek to persuade a public beset by a clear and present danger that air pollution is still the biggest threat they need to worry about.

In any case, the fanfare over the two studies, and the apparent incompatibility of their results, underscores the need for independent evaluation of “pivotal regulatory science”—i.e. “the studies, models, and analyses that drive the magnitude of the benefit-cost calculation, the level of a standard, or point-of-departure from which a reference value [i.e., the safe long-term exposure level] is calculated” (83 FR 18770).

Environmental Protection Network Comment Letter

The Environmental Protection Network (EPN) submitted a 127-page comment letter attacking the Transparency proposal.[17] Among other objections, EPN claims EPA lacks authority to make transparency a criterion for evaluating the science used in regulation,[18] arbitrarily divides regulatory science into two categories (independently validated science for major rules, and non-validated science for permitting decisions),[19] and has not identified the problem the Transparency rule aims to solve.[20]

Contrary to EPN, EPA has authority to elevate independently validated studies in rulemaking.

EPN reviews passages from all eight statutes EPA cites in the Transparency proposal as authority for changing how it evaluates science in rulemaking. EPN contends that because no statute lists “transparency” as a criterion for determining what qualifies as good science, EPA has no authority to make transparency a basis for preferring some studies to other studies.

This discussion looks impressive at first blush but comes to just nothing. Although the statutes repeatedly tell EPA to use the “best available science” or “best peer-reviewed science” or “latest science,” they do not define what science is.

The statutes do not define science because educated people, including lawmakers, are supposed to have some familiarity with the enterprise and how it works. The EPN letter sometimes gives the impression that science is whatever gets published in a peer-reviewed journal. That is not correct.

Science is a mode of enquiry employing a specific method to understand the world. The hallmark of the method is testing hypotheses against data or observations. It is a highly democratic method in the sense that the correctness or incorrectness of a hypothesis has nothing to do with a researcher’s social standing, ideological orientation, or professional credentials. What matters is the agreement or disagreement between hypothesis and data.

Since testing hypotheses against data is the heart of the scientific enterprise, claims that are based on inaccessible data are not really science. A study based on data inaccessible to independent researchers is “trust me science.” 

EPN has no problem with EPA using only peer-reviewed science—and thus, to that extent, “censoring” science. But peer review just means that some scientifically trained people found no obvious errors in a study and think it interesting enough—or politically valuable enough—to publish. As a quality control method, peer review is inferior to post-publication audit by researchers motivated to find flaws in the study, including flaws in its selection and handling of data.

The very nature of science justifies EPA’s preference for studies that can be independently validated and the agency’s attempt to create a set of requirements and incentives that will increase the use of independently validated studies over time.

Contrary to EPN, EPA does not proffer two definitions of science.

According to EPN, the Transparency Rule “applies only to ‘major rules’ and, by a serendipitous and unexplained coincidence, does not apply to many types of decisions where industry is seeking a benefit from the government.”

Specifically, it does not apply to air or water permits for major industrial facilities, to standards for hazardous waste cleanups, to registration or reregistration of pesticides, or the setting of tolerances for pesticide residues on food, or to the approval for market of new chemicals. Such differential application is itself arbitrary.[21]

No, requiring independent validation of all data used in hundreds of decisions regarding permits and product approvals would be quite impracticable, bringing the daily work of the agency to a screeching halt, and imposing unreasonable burdens on the private sector.

Preaching a foolish consistency, EPN argues that “if the agency insists on attempting to promulgate this flawed rule, there should be no distinction made between application of its provisions to major rules and to all other regulatory actions with regard to the consideration of the science in the development of such decisions. If it is appropriate for one category of rules, then it is appropriate for all of them.”[22]

Not if the rules differ in scope, number, and consequence. Consider the National Environmental Policy Act (NEPA). If EPN’s logic were sound, all NEPA proceedings would require full-blown environmental impact statements. There would be no provision for shorter environmental assessments, much less categorical exclusions for actions an agency has determined have no significant impact on the human environment.

Contrary to EPN, EPA has a definite problem it is trying to remedy.

EPN argues that EPA’s consistent practice has been to base regulatory decisions on weight-of-evidence assessments of the peer-reviewed scientific literature. The proposal does not even try to show EPA’s customary approach has “led to bad decisions.”[23] EPN faults EPA for not identifying the problem it is trying to solve. It cites University of Texas law professor Wendy Wagner’s comment in Atlantic Magazine, “I really don’t know what problem they think they are fixing.”[24]

It is a bit much to demand that EPA produce a list of its own “bad decisions.” Each such determination would require a rulemaking of its own. Moreover, the core thesis of the Transparency proposal is not that EPA knows the studies it relies on lead to “bad” decisions but that, in many cases, it cannot properly evaluate those decisions because the underlying studies are incapable of independent validation. Quality control via the “marketplace of ideas” has been precluded.

The problem EPA seeks to address is not obscure, it is obvious: the “replication crisis.”[25] Many studies published in prestige journals turn out to be junk. The Economist, in an editorial cited by the Transparency proposal, explained the problem as follows:

A simple idea underpins science: “trust but verify.” Results should always be subject to challenge from experiment. That simple but powerful idea has generated a vast body of knowledge. Since its birth in the 17^th century, modern science has changed the world beyond recognition, and overwhelmingly for the better.

But success can breed complacency. Modern scientists are doing too much trusting and not enough verifying—to the detriment of the whole of science, and of humanity.

Too many of its findings that fill the academic ether are the result of shoddy experiments or poor analysis (see pages 26-30). A rule of thumb among biotechnology venture-capitalists is that half of published research cannot be replicated. Even that may be optimistic. Last year researchers at one biotech firm, Amgen, found they could reproduce just six of 53 “landmark” studies in cancer research. Earlier, a group at Bayer, a drug company, managed to repeat just a quarter of 67 similarly important papers. A leading computer scientist frets that three quarters of papers in his subfield are bunk. In 2000-10 roughly 80,000 patients took part in clinical trials based on research that was later retracted because of mistakes or improprieties.[26]

The Economist identifies three main causes of replication failure: The publish-or-perish imperative of academic life; the associated career-related inducements to exaggerate, cherry pick results, and spin causality out of chance correlations; publication bias against studies finding negative results; and lax peer review. For example, the article reports, “When a prominent medical journal ran research past other experts in the field, it found that most of the reviewers failed to spot mistakes it had deliberately inserted into papers, even after being told they were being tested.”

In another piece cited in the proposal, economists Randy Lutter and David Zorn similarly underscore the frequency of replication failure:

Over the past few decades, the quality of published scientific research—which underlies most federal efforts to protect consumers generally, and specifically health, safety and the environment—has increasingly come into question. Researchers who tried to replicate the results of peer-reviewed psychology studies succeeded 40 percent of cases. A similar attempt in the field of cancer biology was successful in only 10 percent of cases. When asked, scientists are a little more optimistic. A 2016 survey by the journal Nature found that 73 percent of over 1,500 scientists surveyed believed that at least half of the literature in their field could be independently replicated. In other words, more than one out of four scientists surveyed believed that most of the peer-reviewed literature in their field was not credible.[27]

In an attempted “gotcha,” EPN cites an article titled “All science should inform policy and regulation” by Stanford professor John Ioannidis, the most prominent replication crisis scholar. Ioannidis warns that EPA’s plan to “ban the use of scientific studies for regulatory purposes unless all their raw data are widely available in public and can be reproduced,” would have dreadful consequences: “science would be practically eliminated from all decision-making processes. Regulation would then depend uniquely on opinion and whim.”[28] Two responses are in order here.

First, Dr. Ioannidis’ description of EPA’s proposal is a caricature. EPA is not proposing to “ban” the use of studies unless the raw data are “widely available.” EPA is proposing a system of “tiered” access including the use of highly restricted “data enclaves.”

Independent researchers seeking access to data involving personally identifiable information (PII) would be subject to the same legal requirements and penalties that prevent the original researchers from publicizing confidential information. And that is only if such access is legally permissible. As noted, EPA’s goal is to “ensure that, over time, more of the data and models underlying the science that informs regulatory decisions . . . is available to the public for validation in a manner that honors legal and ethical obligations to reduce the risks of unauthorized disclosure and re-identification” (83 FR 18770).

In the second place, banning studies with inaccessible data would eliminate most science used in rulemaking precisely because of the replication crisis Dr. Ioannidis has documented and brought to national attention. In his words:

Making scientific data, methods, protocols, software, and scripts widely available is an exciting, worthy aspiration. Government-based regulatory and funding incentives can be

instrumental in making this happen at large scale. However, we should recognize that most of the raw data from past studies are not publicly available. In a random sample of the biomedical literature (2000-2014), none of 268 papers shared all of their raw data. Only one shared a full research protocol. The proportion of studies that have had all their raw data independently re-analyzed is probably less than one in a thousand. The number of studies that have been exactly replicated in new investigations is quite larger, but still a minority in most fields.

That is why limiting agency science to independently validated studies would “practically” eliminate science from regulation. EPA is trying to remedy that sorry state of affairs.

It may be many years before a significant fraction of the studies on which EPA relies is sufficiently accessible for independent validation. What should EPA do in the meantime? Lutter and Zorn’s recommendation to former House science chairman Lamar Smith, whose HONEST Act (H.R. 1430)[29] is the legislative inspiration for the Transparency proposal, seems to strike the right balance between boldness and caution:

The bill should also allow agencies to regulate in instances where they do not possess data. Specifically, agencies may rely on research published in peer-reviewed journals, even if there is not public access to such data and the agency cannot acquire it, provided that the agency states it has unsuccessfully sought the data under an agreement providing for privacy of human subjects and protecting trade secrets. In this case the agency, however, would have to state that it is using research based on non-public data and explain why it believes the research is nonetheless sufficiently reliable to be used for regulation.

Respectfully submitted,

Marlo Lewis, Jr., Ph.D

Senior Fellow

Energy & Environmental Policy

Competitive Enterprise Institute

1310 L. Street NW

Washington, DC 20005

202-331-1010

[email protected]