White House Releases a Long Report on Scientific Integrity, One Blind to Its Main Threats

The White House Office of Science and Technology has just released a long 53-page report on the topic of protecting scientific integrity. The report is entitled "Protecting the Integrity of Government Science."  Nowadays in the United States "government science"  means almost the same as "American science," because such a large fraction of research depends on grants from the Federal government. 

The report uses the term "scientific integrity" very many times. But the authors of the report seem to be unaware of the main things undermining scientific integrity these days.  The report focuses mainly on threats to scientific activity such as government interference in scientific activity, such as when an administration or federal agency might try to pressure scientists into reporting something more agreeable to the administration or agency.  The authors of the report make little or no mention of the main things threatening scientific integrity these days. 

Below is a list of the main things threatening scientific integrity these days:

(1) There has arose in scientific journals a tendency called publication bias, a habit of not publishing negative experimental papers reporting only a null result. 

(2) There has arose in academia an incentive system and scoring system in which scientists are judged numerically by the number of scientific papers they have written and the number of citations such papers have received. 

(3) Under such a system there is an incentive for scientists to  produce low-quality papers or low-value papers in high numbers, for the sake of increasing the count of papers they have published, rather than producing papers of much higher quality in smaller numbers. 

(4) Under such a reward system there is also a strong incentive for scientists to run experiments following poor design standards and using inadequate sample sizes, because such easier-to-produce and poorly designed experiments will be more likely to create false alarms that will be reported as some positive result, thereby increasing the chance that the resulting scientific paper will be published, and not be rejected because of publication bias that excludes null results.  

(5) Under such a reward system there is also a strong incentive for scientists to not use proper blinding protocols, and to interpret and analyze experimental data in a biased way, to maximize the chance that some positive result can be reported, rather than a null result (which may cause the resulting paper not to be published because of publication bias). 

(6) Under such a reward system (in which citation counts are a key metric under which scientists are judged) there is also a strong incentive for scientists to exaggerate or misstate their experimental findings or analytic findings, claiming that they showed some important result that was not actually found by the research.  

(7) Wishing to create an aura of research success that increases their institutional prestige, universities and colleges have a strong incentive to write press releases that exaggerate or misstate the research results of scientists at their institutions, making minor or unimportant research sound like some very important result. 

(8) Not wishing to have any scandal that might decrease their institutional prestige, universities and colleges have a strong incentive to not investigate or penalize scientists at their institution who engage in fraud or poor research practices or misstatements about their research. 

(9) Wishing to create additional web traffic that results in more revenue because of online ads that generate revenue proportional to the number of page visitors, science web sites and other web sites have a strong incentive to produce hype-filled misleading pages that inaccurately summarize scientific research, making dubious or unimportant research sound like some important result.  

(10) These incentives are producing exactly the results we should expect them to produce. A large fraction of scientists are producing mainly low-quality or unimportant papers, as if they were more interested in their paper count than in the quality of their papers. In some fields such as experimental neuroscience, very poor research practices are more the norm than the exception, with inadequate sample sizes, a lack of a needed sample size calculation, and nonexistent or inadequate blinding protocols seeming to occur in the majority of experiments.  Very large numbers of scientists are making inaccurate claims in the abstracts or titles of their papers, claiming the research shows things it did not actually show. With great regularity colleges and universities are producing press releases that make inaccurate or exaggerated claims about some research result at their institution that is being announced.  Science news sites and science magazines habitually make unwarranted hype-filled claims about scientific research.  There is a huge replication crisis that has been documented by scientists.  Attempts to replicate experimental results typically show that far fewer than 50% of reported experimental results can be successfully reproduced.  Research surveys of experimental scientists (such as this one and this one) show that a large fraction of them either confess to poor research practices or suspect very many of their colleagues of such conduct.   

(11) In addition to all these problems that have nothing to do with belief traditions among scientists, there are a host of scientific integrity problems resulting from belief traditions that have arisen in scientific communities, cases in which scientists are socially pressured to support or conform to far-from-proven theories that have become popular within scientific communities.  Such theories include the dogma of abiogenesis (that the first living thing arose accidentally), the dogma of common descent (that all species evolved from a common ancestor), the dogma that memories are stored in brains (despite no one ever finding a stored memory in a brain), the dogma that all mental phenomena are caused by brains, the dogma of dark matter, the dogma of dark energy, the dogma of primordial cosmic inflation, the dogma of the nonexistence of spooky psychic phenomena, the dogma that genetically modifying food is safe, the dogma that pesticides are relatively safe (pushed by sites such as www.realclearscience.com), the dogma that gene-splicing is not risky, the doctrine that COVID-19 had a purely natural origin, and many others. When some unproven theory gains ascendancy in a scientific community, and becomes a belief tradition in that community, its members will tend to interpret all observations in a way that conforms to such a theory, rejecting all observations inconsistent with such a theory.  Dogma-driven interpretations of observational results threaten objective scientific inquiry in many ways.   

(12) Because of some of these problems,  a significant fraction of the billions of dollars that the US government spends each year on scientific research is wasted, going to unworthy projects or poorly designed experiments or poorly conducted research.  Moreover, there is a gigantic amount of misinformation being spread about federally funded projects, misinformation such as claims that experiments or projects showed things they did not show. 

The recent White House report on scientific integrity ("Protecting the Integrity of Government Science") seems to be almost entirely blind to such problems.  The authors show no real signs of understanding the problems listed above. Instead of focusing on such problems, which are the main threats these days to scientific integrity, the report focuses on things of lesser significance such as rare cases when some administration attempts to interfere with scientific analysis.  Any report such as this should have said very much about the very widely discussed problem known as the replication crisis, the fact that most scientific research does not seem to be reproducible, and that attempts to reproduce experimental results are failing most of the time. But the 53-page report does not even use the words "replicate" or "replication" or "reproduce" or "reproducible" or "reproducibility."  The report has lots of glittering generalities, but fails to deliver an effective algorithm for how to beef up scientific integrity.  The report has no mention of sample sizes or blinding protocols, two of the main things that should come up in a thorough discussion of scientific integrity. Inadequate sample sizes and a lack of thorough blinding protocols are two of the chief current threats to the integrity of experimental science.  

The report seems to have been carefully worded so that no specific mention would be made of any ongoing shortfall of today's scientists.  It is as if the authors were terrified of offending anyone in the science community or academic community or journalism community.  As a result, the report is a kind of bowl of bland bureaucratic mush, rather than the kind of sharp, incisive thing that might really help beef up scientific integrity.

What might an effective White House report on scientific integrity look like? It might be one that thoroughly documented all the problems threatening the integrity of scientific research.  It might be one that proposed specific measures to substantially reduce such problems.  The report might include a proposal for enforcing quality standards on federally funded research.  Rather than vague bland mush, the report might have included specific recommendations rather like this:

(1) No federal funding of any research project that has its published results available only behind a paywall. Taxpayers should be able to easily read all the research they paid for, except for classified research. 

(2) Some mechanism guaranteeing the publication of all federally funded observational results (including null results). 

(3) Some mechanism (such as public weekly research logs) depriving federally funded researchers from cherry-picking the experimental results that will be published, something that often occurs so that only results conforming to prevailing belief traditions appears. 

(4) No federal funding of any experimental research project unless it first published a detailed research plan precisely describing how data will be gathered and analyzed, and also followed such a plan.  This would mean no more federal funding of "fishing expedition" projects which allow researchers to slice and dice data a hundred ways until they find some result they were hoping to get, and which allow researchers to "torture the data until it makes the desired confession." 

(5) No federal funding of any experimental research project that did not declare before data was gathered a thorough blinding protocol that will be followed, and also adhered to such a protocol.

(6) No federal funding of any experimental research project that did not use at least 15 subjects for each of its study groups. 

(7) A mechanism by which federal money granted for research could be "ungranted," with a refund demanded whenever the standards above were not met. 

If such standards existed for granting federal funding, the US government could stop wasting so many billions on poorly designed experiments producing unreliable or unreproducible results. 

An example of the timidity of the report is the fact that it brings up the topic of what is the definition of scientific integrity, but timidly fails to even define such a thing. We have on page 3 of the report a shaded box labeled "Box 1.1 Defining Scientific Integrity," and we have in that box some vacuous "go around in circles" language:

 

"The 2021 Presidential Memorandum does not define the term 'scientific integrity.' Rather it reaffirms and builds on the 2009 Presidential Memorandum and 2010 OSTP Memorandum, which establish principles and guidance, respectively, for protecting scientific integrity, without explicitly defining the term. The Task Force has taken a similar approach, focusing its initial efforts on assessing agency scientific integrity policies against the principles and guidelines articulated in the memoranda and identifying practices for improving policies and their implementation as called for in the 2021 Presidential Memorandum. The Task Force notes that some, but not all, agencies provide definitions of scientific integrity in their scientific integrity policies. These definitions vary across agencies and would benefit from greater harmonization. The Task Force intends to produce a definition of scientific integrity for adoption by Federal agencies as it develops a framework for assessing scientific integrity policies. The definition will be informed by the insight gained in preparing this report."

This is classic bureaucratic "use lots of words to say nothing" talk.