Scientists Have a Hundred Ways To Conjure Up Phantasms That Don't Exist

Well, that was exciting while it lasted. 

Phosphine is a gas mainly produced on Earth by living things. On September 14, 2020, a scientific paper entitled "Phosphine gas in the cloud decks of Venus" claimed to have detected the "apparent presence" of phosphine in the atmosphere of Venus. Following their long-standing tendency to hype like crazy any story that may serve as clickbait to produce more page views and advertising revenue, a host of web sites began proclaiming that life or a sign of life had been discovered at Venus.  

I may have been the first one in the blogosphere to make a substantive criticism of this claim, since I published on the next day a post entitled "No, They Haven't Detected Life at Venus," in which I cited several reasons for doubting the idea that the paper had provided any evidence of life at Venus.  I said on September 15 that an alternate explanation was that "an error in interpretation could have occurred in spectral data that is hard-to-interpret because of overlapping signals from a variety of different gases in the atmosphere of Venus," and that such a possibility was "not very unlikely." I also pointed out the substantial chance that merely geological processes could have produced phosphine, and pointed out the lack of any plausible scenario for life on Venus, given the incredibly hostile conditions both on its hot-enough-to-melt-lead surface and its clouds (having almost no water but lots of sulfuric acid vapor). 

Days after my post, we began seeing on some sites such as www.liveScience.com and the National Geographic web site some articles (such as this one) questioning whether any sign of life had really been discovered on Venus.  On October 19, 2020 there appeared a scientific paper with the title "Re-analysis of the 267-GHz ALMA observations of Venus: No statistically significant detection of phosphine."  The paper doesn't merely question whether evidence of life on Venus has been found. The paper tells us that no robust evidence of phosphine has been discovered in the atmosphere of Venus. 

The five authors of the scientific paper provide a critique of the  September 14 paper claiming evidence for phosphine in the atmosphere of Venus,  saying that the paper used a dubious statistical technique that "leads to spurious results." The five authors state that the data actually provides "no statistical evidence for phosphine in the atmosphere of Venus." 

If the five authors are right, then we have here another example of what goes on so very often: scientists conjuring up a phantasm by using dubious methods that lead them to suggest the existence of something that does not really exist.  What we should never forget is that when you're a scientist, there are always a hundred ways for you to get some illusory evidence that is just a false alarm.  This doesn't require or typically involve any outright deception. It merely requires for a scientist to use some method that isn't quite right. 

One extremely common way for a scientist to conjure up a phantasm is to use too small a sample size or too small a study group size, which tends to result in false alarms. Scientists know how they can avoid this sin: by doing a sample size calculation to determine the minimum study group size needed to produce a moderately persuasive result, and to only use study groups with such a size. But a large fraction of scientific studies (particularly animal neuroscience studies) fail to include such a calculation, and fail to have adequate study group sizes. 

Another way for a scientist to conjure up a phantasm is to prune or filter his data until the desired thing seems to appear. It might be that when considering his full data set, there will seem to be no evidence of some thing (call it X) that the scientist wants the data to show evidence for.  But the scientist can prune the data at its beginning or end, until some evidence of X seems to show up. For example, if there were 4 weeks of data collection, the scientist can just get rid of the week 1 data or the week 4 data, or maybe both weeks of data. Or, the scientist can apply some "data filter" which gets rids of certain data points, until some evidence of X seems to show up.  The decision to apply such a data filter can often be rationalized in various ways, to make it sound like some "quality filter" excluding "bad data" or "outlier data."

Another way for a scientist to conjure up a phantasm is to collect data in a biased way that will maximize the chance that the desired result will appear. I will give a hypothetical example. Let us imagine that you are a scientist who wants to show that rainy days in New York City cause a higher chance of 300-point drops in a stock market indicator such as the Dow Jones Industrial Average.  You might begin recording daily stock market results on a rainy day in which there was a 300-point drop in the stock market.  You might then continue to record daily results, and conveniently end your data collection on a rainy day in which there was more than a 300-point drop in the stock market.  Given such convenient start and stop points of your data collection, you may well be able to write up a "statistically significant" correlation between rainy days in New York and 300-point drops in the stock market. But if you had resolved beforehand to start collecting data on some day 14 days in the future, and continue collecting data for exactly 100 days, then the desired result would probably not show up. 

Once so-called "raw data" has been collected, there are 1001 ways to "massage" the data before it is analyzed, some of which may make sense and some of which are dubious. A scientist can produce all kinds of rationalizations for particular data exclusions and data inclusions and data averagings and "data smoothings" and "weighted averagings" that may have been used, which can have a huge effect on whether some illusory phantasm shows up. 

Moving from the topic of data collection to data analysis, there are innumerable ways in which a scientist can conjure up phantasms by some kind of data analysis that isn't quite right. Don't be reassured when some science paper claims that it uses some kind of "standard software" for data analysis. There is almost always no such thing as a "standard analysis" of data. There are standard software tools used for data analysis, but such tools can be used in a million valid ways, and a million dubious ways. An example is Microsoft Excel, the leading spreadsheet program.  There are a million bad ways to use it, as well as a million good ways. 

A science paper may try to reassure you that it used some standard software for doing some type of data analysis (such as measuring brain scan data or trying to measure "freezing behavior" in mice).  But there are always countless different ways to use such software, some good and some bad.  Every software program has program settings or startup options or menu options that allow you to customize how the program is used. Software programmers usually ask "how can I give the user the freedom to do exactly what he wants," and almost never ask "how I can make it so that there's no way to use the software in a stupid way." 

Another way a scientist can conjure up phantasms is by failing to do a preregistered study that announces an experiment will be testing one very specific hypothesis, and going on a kind of "fishing expedition" within his analytic activity. For example, let's imagine the scientist does brain scans looking for some correlation between some behavior (or some aspect of thinking) and activity in some brain region. By failing to limit himself to checking one small specific part of the brain corresponding to a previously declared hypothesis, and giving himself the freedom to check any of 100 small parts of the brain, he will have a good chance of finding some tiny region that weakly correlates with the behavior or aspect of mentality.  That's simply because given 100 parameters that show random variations, and the freedom to check any of them, it's easy to get something that looks like a slight correlation, even if only chance and not causation is involved.  The name sometimes given for this procedural sin is HARKing, which stands for Hypothesizing After Results are Known. 

Then there's the matter of what the scientists wishes to call attention to in writing up the paper.  Examining exactly the same data, one scientist may choose to make some little data anomaly get special treatment by referring to it in the paper title, while another scientist may "bury" such a data anomaly by mentioning it only in a footnote or in the paper's "supplemental information."  

Or a scientist can conjure up some phantasm by simply having the paper announce something was found that was not actually found. This is a surprisingly common sin. A scientific paper found that 34% of scientific papers "used language that reviewers considered too strong for their strength of causal inference."  Such unfounded claims often occur in the title of scientific papers, which often proclaim some result that is not justified by anything in the paper. 

A scientist may conjure up a phantasm by carefully cherry-picking some examples supporting some speculative idea, while conveniently failing to mention other examples that argue against such an idea. I have recently read two scientific papers that mentioned some evidence for some speculative idea, and conveniently failed to mention the largest studies done on the topic, which presented data against such an idea. 

A scientist may conjure up some phantasm by using some loaded term that isn't appropriate. For example, he may refer to some brain region being "activated" when some mental event occurs, even though the brain region is actually constantly active, and is merely showing something like a 1% greater-than-average activity, something that will randomly occur in different brain regions when nothing special is occuring.  Or the scientist may refer to a "gene regulatory network," rather giving you the idea that genes act like some committee members making decisions. Genes are mere mindless low-level chemicals.  Or the scientist may claim that some behavior is caused by "hard-wiring" in the brain, even though no one has ever found any brain structure (other than reflexes) that determine behavior. 

The things mentioned above are generally not clear-cut deceptions. But it does sometimes happen that scientists conjure up a phantasm by simply using language that is just plain false.  For example, many a scientist has spoken of DNA as being a blueprint or recipe for making a human. Containing only low-level chemical information and no instructions for making anatomy, DNA is no such thing, and bears no resemblence to either a blueprint or a program or a recipe for making a human or any other organism.  "DNA as a blueprint for organisms" is one of the most notorious phantasms of scientist literature. 

Scientists have a hundred ways to conjure up illusory phantasms, and once a phantasm has been conjured up, there are many tricks by which the phantasm may be made to seem like something real, such as the use of complex charts and thick jargon which make the problematic presentation seem very scientific. All in all, we may say that the power of scientists to give you an impression of the reality of something illusory is comparable to the similar power of Hollywood's CGI special effects wizards. 

Postscript: Now we have two additional papers saying that there is no phosphine in the atmosphere of Venus. One paper by a single author states, "There is thus no significant evidence for phosphine absorption in the JCMT Venus spectra." Another paper with many co-authors is entitled, "No phosphine in the atmosphere of Venus." 

The case of the "phosphine at Venus" error might give you the impression that scientist errors are quickly corrected. That sometimes happens, but other scientist errors may become enshrined and persist for decades or centuries.  A particularly notorious example is the case of the fraudulent or erroneous embryonic drawings made by biologist Ernst Haeckel.  The drawings were made to try to give evidence for the idea that early stages of the human form resemble animals from which humans were originally descended.  For well over 130 years, the bogus drawings were some of the chief evidence cited for the doctrine of common descent, the idea that all organisms have a common ancestor. You may still find these bogus drawings in evolutionary textbooks at your local library.