Why It's Silly to Disparage Anecdotal Evidence

My most recent post was a science fiction story which took a jab at the strange thought patterns of some of today's scientists. In the story, invading extraterrestrials arrive near our planet, and start leaving more and more dramatic evidence of their arrival. The scientists advising the US president keep denying that anything paranormal is going on, despite the evidence piling up to extremely high levels. The US president is lulled into inaction by these assurances, and the chance to defend against the attack is lost, resulting in the conquest of planet Earth.

At one point in the story, a scientist says this in response to massive testimony about extraterrestrial activity:

You can't trust anecdotal evidence like that. We don't trust the reports of ordinary people. Our motto is: only believe it if it was reported in a scientific journal.

Now, upon reading that, you might complain that this is a crude caricature of what scientists say. But two days after writing the story, I read a scientist say almost exactly this. The scientist (Alex Bezerow) makes this claim:

Testimonials mean nothing. Period. Anybody with a modicum of scientific training understands that.

The scientist then goes on to offer what he claims is an “excellent example of how dangerous and misleading anecdotal evidence can be” by citing a debunking of the famous Erin Brokovich lawsuit, a debunking which is itself debunked in this article.

Bezerow's claim that “testimonials don't matter” is a wholesale dismissal of all anecdotal evidence. This is nonsensical. Anecdotal evidence is simply evidence in which a person describes what he did, or what happened to him. Such evidence is a key pillar of our legal system, and (as I will show in a moment) is actually a key pillar of modern science.

When someone makes a ludicrous claim such as “testimonials mean nothing,” or claims that anecdotal evidence doesn't count, you have to ask: what is their motive? What would motivate someone to advance such a bizarre principle, so contrary to common sense?

I can think of two motives that a scientist might have for advancing such a principle. The first is that a dismissal of anecdotal evidence is very convenient for someone who wishes to advance a narrow, restrictive view of reality, particularly a view dominated by mechanistic or materialist principles. If you want to maintain that there is nothing but matter and energy, you have the problem that a significant fraction of the population report psychic, paranormal, or spiritual experiences that go beyond such a limited reality. It is very convenient, therefore, to be able to “cross out” all such reports by claiming that anecdotal reports aren't good evidence.

Another motive that a scientist might have for advancing such a principle is that he might be trying to shore up scientists' attempts to establish a kind of knowledge monopoly, in which scientists are regarded as some special priests of learning who are the sole possessors of the keys to truth. Such attempts have grown more and more brazen in recent decades. What better way to help enthrone our scientists as the sole judges of truth than to tell the lowly masses that what they observe and experience doesn't count, that only the work of scientists counts towards establishing the truth?

But such attempts are futile. One reason is that a large fraction of our scientific papers are themselves anecdotal evidence.

Consider a scientist who does experiments with rats or chemicals, and publishes the results in a scientific journal. The account of the experiment is itself anecdotal and testimonial. So if we disregard anecdotal evidence, we must then disregard all of the scientific papers in which individual scientists describe particular things they did in an experiment. Bang, you've just wiped out a large fraction of modern science.

It is futile to rebut this objection by saying, “Such evidence isn't anecdotal-- it's experimental.” A typical account of an experiment that a scientist performed is both experimental and anecdotal, so you don't show it's not anecdotal by showing it is experimental.

It's also futile to rebut this objection by claiming that when a scientist makes a claim in a scientific journal about his experiences, that this has more weight than a non-anonymous account by an ordinary person, on the grounds that the scientific journal is peer-reviewed. When a scientific paper is peer reviewed, the reviewers do not question the author, and do not ask to see corroborating evidence.  So the testimonial report of a scientist as to his experience while running an experiment has no more weight than the typical published account of a non-scientist who gives his correct name.  A scientist doesn't sprinkle some magic truth dust on his anecdotal account by publishing it in a journal. 

Of course, scientists don't actually follow the principle that “testimonials don't matter” or that “anecdotal evidence doesn't count.” But a scientist may occasionally evoke such a principle when he or she finds it convenient to dismiss some evidence that he doesn't wish to accept.

We can imagine how following the “ignore anecdotal evidence” principle might lead to the unnecessary death of thousands. A pharmaceutical company might introduce a new drug with baleful side effects. Many people might report that their spouses died or had a stroke after taking the drug. Such reports could be ignored as mere “anecdotal evidence.” The death toll might mount, until tens of thousands died. Finally, lagging far beyond the anecdotal evidence, some scientific studies might show the drug was dangerous, and the drug might be withdrawn by the pharmaceutical company.

This scenario isn't imaginary. It's the actual history of a drug called Vioxx, which is estimated to have killed 38,000 people.