He's Apoplectic That This Anomaly Is Being Researched

As shown in this post, the ire of neuroscientist Steven Novella was recently ignited. Very strangely the bitter indignation of this scientist has been kindled by the simple fact that a research laboratory has been opened.

The research laboratory is a laboratory in India to study the anomaly known as homeopathy. I have never tried homeopathy, and have never recommended it. But I know that it is an alternative medical technique that can sometimes involve giving people extremely diluted solutions. A believer in this technique may believe that if you take a certain type of concentrated solution, and then dilute it by a factor such as 10,000 times or more, the solution will still have some medical potency (and the medical potency can still be retained even if there is no detectable trace of what was originally in the concentrated solution).  

Based on what we currently know about chemistry, you would expect homeopathy to have no effectiveness whatsoever. But surprisingly, research studies have often seemed to show real medical effectiveness from such extremely diluted solutions. There seem to be three possibilities here:

1. Homeopathy has no real effectiveness, and studies suggesting otherwise are just false alarms.
2. Homeopathy does have some real effectiveness, and its effectiveness is because nature has some important aspect that our chemists have overlooked or not yet discovered.
3. Homeopathy can sometimes be effective not for chemical reasons but simply because some people believe it is effective; and homeopathy is a case of a mind-over-matter effect or a placebo effect in which a person's expectations or beliefs can affect his medical outcomes. (Notably, a meta-analysis in the British medical journal Lancet found that homeopathy produced results substantially better than could be explained by a placebo effect; it stated, "The results of our meta-analysis are not compatible with the hypothesis that the clinical effects of homoeopathy are completely due to placebo.")

Because items 2 and 3 are of very significant scientific and intellectual interest, it seems that homeopathy is worthy of further study and investigation. So I am puzzled by the irate reaction of Steven Novella to an Indian news story that merely mentions that a new research center has been opened in India to study homeopathy, without even making any general claims about homeopathy. Why would a scientist not want an anomaly to be investigated further? Could it be that Novella is worried that the research might find something that challenges his dogmatic proclamations on the topic of homeopathy?

Novella has a link saying “homeopathy does not work for anything.” When I follow that link, it takes me to another post by Novella mentioning mainly the NHMRC report on homeopathy. In a previous post, I thoroughly examined this report, and found it to be a prime example of a faulty and biased meta-analysis. I documented quite a few defects in the meta-analysis, such as arbitrarily excluding studies with fewer than 150 subjects, a cut-off level that is not typically used by other medical meta-analyses. A typical meta-analysis on a topic other than homeopathy will include studies having 75 or 100 subjects.  The NHMRC considered only 225 research studies out of more than 1800, an exclusion rate far higher than we have with other similar meta-analysis studies. There are guidelines called the PRISMA guidelines which give recommendations on how a meta-analysis should be done. The NHMRC report violated such guidelines. So we cannot use the NHMRC as a guide to whether homeopathy is effective, and its meta-analysis does not cross-out the Lancet meta-analysis suggesting that homeopathy has an effectiveness better than a placebo. 

Novella also compares ESP and homeopathy, and the information he gives on ESP is dead wrong. He says this:

After a century of research and thousands of studies there is no clear evidence that ESP is real. For both homeopathy and ESP there is a great deal of noise, but no clear signal. There are many flawed or small studies, but no repeatable high quality studies.

What Novella is telling us about ESP is entirely wrong. Sound experimental research showing the reality of ESP has been done for more than 130 years. Among the research highlights was the work of Professor Joseph Rhine at Duke University. Under controlled experimental conditions, Rhine showed spectacular results such as tests showing 3746 successes out of 10,300 tries (a 36% success rate), in experiments in which the expected success rate was only 20%. We would not expect such a result to occur by chance even if every person on planet Earth was tested for ESP. The subject in question (Pearce) got even better results testing with another researcher (Pratt), getting 558 successes in a card-guessing experiment in which the expected number of successes was about 370. Such a success rate occurring by chance had a likelihood of less than 1 in 10,000,000,000,000,000,000,000. Pearce's successes were repeated, showing Novella's claim about repeatibility is inaccurate. An even more spectacular result was reported by Professor Riess, who did a remote card-guessing test showing a success rate of 73% on 1850 guesses in an experiment in which the expected success rate was only 20%.

More recent research on ESP has included sensory deprivation experiments called ganzfeld experiments. Done by many researchers using many subjects, such experiments have repeatedly  shown success rates of 30% to 32% and higher on tests in which the expected chance rate is only 25%. This is a very high degree of repeatability. Even more dramatic recent results are summarized at the end of this post. There are also innumerable very dramatic  anecdotal reports of ESP collected by researchers such as Louisa Rhine, and summarized in the book The Gift. 

So Novella is simply misinforming us about ESP. He's told us that “there is no clear evidence that ESP is real,” which is false. He's also told us that there “are no repeatable high quality studies,” but there are very many such studies, and ESP is very much an effect that shows up dramatically in repeated scientific studies.

Given that Novella has misinformed us about ESP, we may wonder whether he is also misinforming us about the evidence for homeopathy. He tells us that there is in regard to homeopathy a “great deal of noise, but no clear signal,” and if you read between the lines, that “great deal of noise” sounds like something that could be real evidence of an important reality behind homeopathy. How to sort out whether there is real evidence? One way is to do more research. So Novella's ire about a homeopathy research center appearing is strange, and also unscientific.  

 "Don't bother me with more data" isn't scientific

Postscript: Another example of a Novella misstatement is this  post with the title, "AWARE Results Finally Published -- No Evidence of NDE." The AWARE study (which you can read about here and here) actually found very dramatic evidence for near-death-experiences (NDE), including a case of a man reporting floating above his body while his heart had stopped (including independent verification of his recollections of his medical resuscitation efforts while he was unconscious), and a person who reported traveling through a tunnel toward a very strong light, and encountering a beautiful crystal city, along with about 100 other near death experiences.

Postscript: Novella's rage against paradigm-challenging research continues in this post, where he fulminates against the existence of the Susan Samueli Center for Integrative Medicine.  In the post Novella claims that acupuncture "does not work." But a New York Times article asserts otherwise, saying the following:

A new study of acupuncture — the most rigorous and detailed analysis of the treatment to date — found that it can ease migraines and arthritis and other forms of chronic pain. The findings provide strong scientific support for an age-old therapy used by an estimated three million Americans each year.  

In the post, a methodologist at the very prestigious Memorial Sloan-Kettering Cancer Center in New York says, "We think there’s firm evidence supporting acupuncture for the treatment of chronic pain."

How Not To Do a Meta-Analysis

I have no idea whether the esoteric practice known as homeopathy has any medical effectiveness, and this will certainly not be a post intended to persuade you to use such a practice. I will be examining instead the unfairness, methodological blunders and sanctimonious hypocrisy of an official committee summoned to convince you that homeopathy is unworthy of any attention. Committees such as this are part of the reality filter of materialists, the various things they use to try to cancel out, disparage, or divert attention from the large number of human observations inconsistent with their worldview. 

 

 

Issuing a 2015 meta-analysis report on homeopathy, the committee called itself the Homeopathy Working Committee, and was sponsored by or part of the the National Health and Medical Research Council (NHMRC), an Australian body. The committee consisted of 6 professors and one person who was identified merely as a consumer.

On page 14 of the committee's report, the committee makes this confession: “NHMRC did not consider observational studies, individual experiences and testimonials, case series and reports, or research that was not done using standard methods.” What kind of unfairness is that? Using such a rule, if a committee investigating a medical technique were to receive a million letters saying the technique produced instantaneous and permanent cures of dire maladies, the committee would just discard all such letters and not let them influence its conclusion.

The committee limited itself to scientific studies, but it did not at all simply consider all of the scientific studies on homeopathy. Instead, the committee chose to disregard the vast majority of scientific studies on homeopathy, and consider only a small subset of those studies. This is made clear by a Smithsonian article on the committee's report, which says, “After assessing more than 1,800 studies on homeopathy, Australia’s National Health and Medical Research Council was only able to find 225 that were rigorous enough to analyze.” But what was going on was actually this: the committee cherry-picked 225 studies out of more than 1800, claiming that only these should be allowed to influence its conclusions. So it based its findings on only about 12 percent of the total number of scientific studies on the topic it was examining, excluding 88% of the studies. I have never heard of any meta-analysis that excluded anything close to such a high percentage of the studies it was supposed to be analyzing.

The committee claimed to have used quality standards, standards that relatively few of the studies met. What were these standards? Below is a quote from the committee's report.

The overview considered only studies with these features: the health outcomes to be measured were defined in advance; the way to measure the effects of treatment on these outcomes was planned in advance; and the results were then measured at specified times (prospectively designed studies); and the study compared a group of people who were given homeopathic treatment with a similar group of people who were not given homeopathic treatment (controlled studies).

It is not at all true that medicine or science has these criteria as standards that are followed by all or even most studies. Control groups are when you have some people studied who are not subject to what is being tested in another group. A large fraction of all scientific studies and medical studies do not use control groups, for various reasons. Controls are often not practical to implement, too expensive to implement, or not needed because it is clear what the result will be in the case of zero influence. This scientific paper says the following about control groups:

The proportion of studies that have control groups in the ten research domains considered range from 3.3% to 42.8% ..Across domains, a mere 78 out of the 710 studies (11%) had control groups in pre-test posttest designs.

It also is extremely common for medical and scientific research to report findings that the study was not designed to look for. Saying that a study must only report what it was designed to measure is a piece of sanctimonious rubbish, rather like claiming that good students must only get ancient history answers by reading the original ancient texts, rather than looking up the answers on the Internet. Under such a rule, we would for example ignore very clear findings that homeopathy was effective in reducing arthritis pain, if the study was designed to look for whether homeopathy was effective in reducing headaches. I have never heard of any meta-analysis excluding studies based on whether they reported unexpected findings the study was not designed to look for. This seems to be an aberrant, non-standard selection rule.

So what we have here is a committee using a double standard. It has declared that scientific studies will not be considered unless some particularly fussy standard is met, a standard that a large fraction of highly-regarded scientific studies do not meet. It's like the door guard of the country club saying “Ivy league graduates only” to dark-skinned people trying to get in, even though he knows he just admitted some white people who don't even have college degrees.

The statement below from the committee's report also is a sign of double standards and cherry-picking.

For 14 health conditions (Table 1), some studies reported that homeopathy was more effective than placebo, but these studies were not reliable. They were not good quality (well designed and well done), or they had too few participants, or both. To be confident that the reported health benefits were not just due to chance or the placebo effect, they would need to be confirmed by other well-designed studies with adequate numbers of participants.

On page 35 we learn that the actual participant size requirement used by the committee was a minimum of 150 participants (studies with fewer participants were ignored). So if there had been 500 studies each showing that between 110 and 149 patients were instantly cured of terminal cancer, such studies would all have been excluded and ignored. How silly is that? For comparison, a meta-analysis on stuttering treatments excluded only studies with fewer than 3 participants; a meta-analysis on diabetes excluded only studies with fewer than 25 participants; and a cardiology meta-analysis included studies with as few as 62 participants.

I very frequently read about scientific studies which used only a small number of participants (30 or less), studies getting lots of coverage in the media, after being published in scientific journals. So evoking “too few participants” as an exclusion criteria (based on a requirement of at least 150 participants) is another example of a double standard being used by the committee. And once a committee has declared the right to ignore any study that does not meet the vague, arbitrary, subjective requirement of being “good quality (well designed and well done)," it has printed itself a permission slip to ignore any evidence it doesn't want to accept.

Below is a page from a statistician's presentation on whether or not small sample sizes should be excluded when doing a meta-analysis of medical studies. The recommendation is the opposite of what the homeopathy study committee did.

Similarly, the “Handbook of Biological Statistics” site says, “You shouldn't use sample size as a criterion for including or excluding studies,” when doing a meta-analysis.

In the case of homeopathy, it's particularly dubious to be excluding small studies with less than 150 participants. Only a small fraction of the population believes in the effectiveness of homeopathy. It is entirely possible that because of some “mind over body” effect or placebo effect, homeopathy is actually effective for those who believe in it, but ineffective for those who don't believe in it. So we are very interested in whether it is effective for small groups such as a small group that believes in homeopathy. But we cannot learn that if a committee is arbitrarily excluding all studies with fewer that 150 participants.

No doubt if we were to examine the scientific papers of the professors in the committee, we would find many that had the same issues of small participant size or no control groups or reported effects that the study was not designed to show (or we would find these professors had authored meta-analysis papers that included studies that lacked one or more of these exclusion criteria). So it is hypocrisy for such a committee to be using such things as exclusion criteria.

Apparently the committee used some type of scoring system to rate studies on homeopathy. One of the subjective criteria was “risk of bias.” We can guess how that probably worked: the work of any researcher judged to be supportive of homeopathy would be assigned a crippling "risk of bias" score making it unlikely his study would be considered by the committee. But what were the scores of the excluded studies, and what were the scores of the studies that were judged to be worthy of consideration? The committee did not tell us. It's kept everything secret. The report does not give us the names of any of the excluded studies, does not give us URL's for any of them, and does not give us the scores of any of the excluded studies (nor does it gives the names, the URLs or the scores of any of the studies that met the committee's criteria). So we have no way to check on the committee's judgments. The committee has worked in secret, so that we cannot track down specific examples of how arbitrary and subjective it has been.

There is a set of guidelines for conducting a medical meta-analysis, a set of guidelines called PRISMA that has been endorsed by 174 medical journals. One of the items of the PRISMA guidelines is #19: “Present data on risk of bias of each study and, if available, any outcome level assessment.” This standard dictates that any subjective “risk of bias” scores used to exclude studies must be made public, not kept secret. The NHMRC committee has flaunted such a guideline. The committee has also ignored item 12 on the PRISMA guidelines, which states, “Describe methods used for assessing risk of bias of individual studies.” The NHMRC committee has done nothing to describe how it assessed a risk of bias. Nowhere do the PRISMA guidelines recommend excluding studies from a meta-analysis because of small sample size or whether the reported effects match the effects the study was designed to show, two aberrant criteria used by the NHMRC committee.

It has been recommended by a professional that whenever any meta-analysis uses a scoring system to exclude scientific studies on the topic being considered, that such a meta-analysis should always give two different results, one in which the scoring system is used, and another in which all of the studies are included. That way we could do a sensitivity analysis in which we can see how much the conclusion of the meta-analysis depends on the exclusion criteria. But no such thing has been done by the committee. They have secretively kept their readers in the dark, by revealing only the results obtained given all of their dubious exclusions.

After doing all of this cherry-picking based on double standards and subjective judgments, the committee reaches the conclusion that homeopathy is not more effective than a placebo. But even if such a thing were true, would that make homeopathy worthless for everybody? Not necessarily.

Here's the story on placebos. Placebos have repeatedly been shown to be surprisingly effective for certain conditions. A hundred years ago, your doctor might have given you a placebo by just handing you a bottle of sugar pills. But nowadays you get your medicine in labeled plastic containers at the pharmacy, and people can look up on the Internet anything that is on the label. So a doctor can't just write a prescription for sugar pills without the patient being able to find out it's a placebo. But if a patient thinks some particular thing will work – homeopathy, acupuncture, holding a rabbit's foot, praying, or meditation – that might act as a placebo with powerful beneficial effects. 

We therefore cannot dismiss something as being medically ineffective by merely saying it's no better than a placebo. Imagine there's a patient who doesn't trust pills, but who tends to believe in things like homeopathy. Under some conditions and for certain types of patients, homeopathy might help, even if what's going on is purely a “mind over body” type of placebo effect, rather than anything having to do with what is inside some homeopathic treatment.

If there are “mind over body” effects by which health can be affected by whether someone believes in a treatment, such effects are extremely important both from a medical and a philosophical standpoint, since they might be an indicator that orthodox materialist assumptions about the mind are fundamentally wrong. Anyone trying to suppress evidence of such effects through slanted analysis shenanigans has committed a grave error.

Based on all the defects and problems in this committee's report, we should have no confidence in its conclusion that homeopathy is no more effective than placebos; and even if such a conclusion were true it would not show that homeopathy is medically ineffective (since placebos can have powerful medical effects). `The fact that 1800 studies have been done on homeopathy should raise our suspicions that at least some small subgroup is benefiting from the technique. It doesn't take 1800 studies to show that something is worthless – one or two will suffice.

Whether homeopathy has any medical effectiveness is an unresolved question mark, but about one thing I am certain. The committee's report is an egregious example of secretiveness, double-standards, overzealous exclusions, guidelines violations, and sanctimonious hypocrisy. Using the same type of methodological mistakes, you could probably create a meta-analysis concluding that smoking doesn't cause lung cancer; but you would mislead people if you did that. 

Postscript: Today's New York Times criticizes "the cult of randomized controlled trials" and points out the case of those who say the evidence for the effectiveness of flossing is weak, because there aren't enough randomized controlled trials showing it works. That, of course, makes no sense, as we have abundant anecdotal evidence that flossing is effective -- just as we have abundant evidence that parachutes work, despite zero randomized controlled trials showing their effectiveness. 

Postscript: A meta-analysis was recently published on the effectiveness of homeopathy in livestock. The meta-analysis avoided the outrageous exclusion problems discussed above; for example, it didn't exclude studies based on sample size. The meta-analysis concluded, "In a considerable number of studies, a significant higher efficacy was recorded for homeopathic remedies than for a control group." Specifically it concluded that "Twenty-eight trials were in favour of homeopathy, with 26 trials showing a significantly higher efficacy in comparison to a control group, whereas 22 showed no medicinal effect."  What is astonishing is that this result favoring homeopathy has been reported in The Scientist magazine with the headline, "Homeopathy does not help livestock."  That's the opposite of what the meta-analysis actually found.