The Glaring Defects in the Dutch Hallucination Study

Skeptics of the paranormal have a strong motivation to misstate or misrepresent the number of people who have hallucinations. If a person can somehow show that ordinary sane people without mental problems frequently have hallucinations, then it may be easier to explain away frequent observational reports in which people report seeing or hearing things that scientists cannot explain. A recent Dutch study claims to provide evidence that most people have hallucinations. The study has not actually found any good evidence for such a claim. The methodology of the study is apallingly bad. 

A misleading press release by the British Psychological Society  announces some untrue claims about the study.  The press release claims this: "About 80% reported having hallucinated at least once in their lifetime." That is not correct.  The study did not ask people to specify whether they had hallucinated. Instead, the study used an online survey containing ridiculously loaded questions that seemed designed to get as many people as possible to give answers that could be judged as evidence of a hallucination. 

The English-language paper presenting the study has done a great job of hiding the loaded questions asked by the survey. No English language reader can discover the questions used by simply reading the paper.  The paper authors offer to give an English translation of the Dutch language questions they used in the survey. But using the paper, I am unable to discover the email addresses of any of the authors. There's an email icon next to one of the authors, but clicking on that icon does not give me any email address. There are some links to profiles of one of the researchers, but none of the profiles contains an email address. 

You have to work rather hard to find out the questions that were used, by using this technique:

(1) First you can download the supplemental information document contained here, which contains the questions only in Dutch. 

(2) Then you can use the Google Translate facility to translate those questions from Dutch to English. 

Upon doing that, you will discover how ridiculously loaded the questions were. They were as loaded as some district attorney asking a defendant, "So how did you feel when you committed the crime -- calm or angry?" For example, here is the first question asked (I use the Google Translate translation):

"1. It sometimes happens that one hears someone speak, while there seems to be no one there. Also can sounds or music be heard, although it is not clear where they come from... Have you ever heard such voices, music or other sounds?

0: no

1: yes

If so, have you experienced this in the past week?

0: no

1: yes"

It is preposterous to be asking such a question as a way of trying to determine how many people have auditory hallucinations. There are all kinds of natural, mundane, ordinary reasons why a person might hear someone speak while there seems to be no one there. It is also extremely common for sounds or music to be heard, even though it is not clear where they come from. It is ridiculous to be judging the frequency of auditory hallucinations from the number of people answering "yes" to the question above. 

The second question in the survey has an equally preposterous wording. The translation I get using Google Translate is below. The translation of "do you ever have such things seen people or images" presumably is an imperfect translation of some phrase that meant something like "have you ever seen such people or images":

"2. It sometimes happens that one sees a person, animal or thing that others cannot...see. Some people sometimes see a shadow or shadow. Do you ever have such things seen people or images?

0: no

1: yes

If so, have you experienced this in the past week?

0: no

1: yes"

It is preposterous to be asking such a question as a way of trying to determine how many people have visual hallucinations. There are all kinds of natural, mundane, ordinary reasons why some person  might see a person, animal or thing that others cannot. It is ridiculous to be judging the frequency of visual hallucinations from the number of people answering "yes" to the question above. It seems almost anyone who had ever seen a shadow would answer "yes" to such a question, and would be senselessly judged by the survey as having had a visual hallucination. 

It is rather obvious from the two questions above that the survey was designed to produce the largest possible number of "yes" answers, so that some super-inflated estimate could be made of the number of people having hallucinations. From such a survey nothing reliable can be deduced about the percentage of people who have hallucinations. 

Besides the ridiculous wording of the questions, so prone to extract "yes" answers from normal people who are not hallucinating, the methodology of using a voluntary online survey was an absurd approach to determining the percentage of people having hallucinations.  A reasonable technique would be to do a random sample of the population, using a randomly selected group of subjects. Instead, the project used a nationally promoted voluntary-participation survey entitled "Do I see ghosts?" Of course, with such a survey title and voluntary participation, the type of people who will be most likely to participate are those who think they have had anomalous perceptions. But getting a certain response rate from such a group of people does nothing to show that such a response rate would occur in the general population.  

I remember a time more than 30 years ago when I worked two full-time jobs for a period of months. My second full-time job was a temporary job working for the US Census Bureau in Boston. The US government wanted to find out what percentage of the population used the fishing and wildlife services supported by the US government.  Workers like me were given stacks of survey forms, each of which had the name of a randomly selected US citizen. My job was to call up such people, and insist that they answer the survey's questions over the phone, questions asking about how often they used the government-supported fishing and wildlife facilities. I would have to keep calling back later if someone claimed to be too busy to answer when I called.  I would very frequently get responses like this from annoyed people:

"Why are you bothering to ask me about such things? I have never gone fishing in my life, nor have I ever gone hunting. So the government shouldn't be asking me about such things!"

I had to explain to such people the concept of a survey of random people: that the only scientifically valid way to find out what percentage of people used the government's fishing and wildlife facilities was to ask randomly selected people about this topic, to keep asking until all of them answered the questions, and to be just as interested in getting "no" answers as "yes" answers.  The US Census Bureau knew how to do a scientifically valid survey. The people at that bureau knew that it never would have been valid to just advertise some survey about fishing and wildlife, and to record what percentage of people choosing to do the survey said that they used fishing and wildlife facilities.  If you did the survey that way, it might have been that most of the participants would have been those who loved to do fishing and hunting.  The survey might then have given very misleading results perhaps suggesting that most people in the US use the government's fishing and wildlife facilities, when in fact only a small minority of the population used such facilities.  

Too bad our Dutch researchers acted as if they did not seem to have the same knowledge of the proper way to determine what percentages of the population have a particular kind of experience. You cannot find out what percentage of the population has some experience by advertising some voluntary survey that will tend to attract only people who have had that type of experience. For example, if you advertise a voluntary "Do you see Jesus in your toast?" survey, you may get mainly people who think they have seen Jesus in their toast.  70% of the respondents may say they saw Jesus in their toast, even though fewer than 1% of the population claims to see Jesus in their toast. 

We should be deeply dismayed by the British Psychological Society's promotion of this junk research. We should also note the "hide the dirty linen" behavior of the paper authors, who made it very hard for English language readers of an English language paper to discover the appalling choice of questions used in the survey.  Advertised to attract participants, the voluntary survey attracted more than 10,000 participants. We may only wonder how many hundreds of normal people who never hallucinated have got the false and psychologically damaging idea that they have hallucinated after reading press reports about this Dutch study after participating in it, and after hearing how people who answered affirmatively to its questions were branded as people who were hallucinating. 

No one could ever get even a ballpark idea of how many people  hallucinate by doing surveys anything like the Dutch survey. The only way to get a reasonably accurate idea of what percentage of people hallucinate would be do something like trying to determine what percentage of people are being treated for hallucinations by professionals such as psychiatrists, or who are identified by friends or family members as someone who hallucinates. Any such effort would suggest that only a very tiny fraction of the population has hallucinations.

It makes no sense to try to debunk the paranormal by doing some survey trying to show that large fractions of the population reports seeing things or hearing things that mainstream science has trouble explaining. For example, if you do a survey showing that 60% of people see things in the sky they cannot explain, that is a stronger reason for suspecting UFOs exist, not some evidence against UFOs. And if you do a survey showing that 50% of people report hearing the voices of their dead parents, that would be a stronger reason for suspecting that communication can occur from the deceased, not something arguing against such a possibility. It is topsy-turvy upside-down "ass-backwards" kind of reasoning to be suggesting that the more people report seeing something or hearing something, the more we should exclude it as a possibility or a reality. 

I read in a medical paper that the mere belief that you have psychiatric problems may cause a higher chance of suicide. A paper notes, "There is an emerging literature to support the hypothesis that stigma variables contribute to suicidality." Some people have almost made a career out of trying to paint observers of the paranormal as people having hallucinations or people with psychiatric problems, such as a certain British professor who can be called "the king of gaslighting." Skeptics of the paranormal are engaging in potentially destructive business when they go about trying to paint normal people as those who have had hallucinations. Doing such a thing can be like shooting a poison arrow in someone's direction.