A Consensus of Experts Means Little, But Evidence Means a Lot

Some people try to make a very simple argument to get you to believe something: believe in some claim, because there is a consensus of experts saying the claim is true. But you should reject such arguments. The mere fact that a consensus of experts supports some claim does not show the likelihood that such a claim is true.

One very general reason why a consensus of experts means very little is that experts may have a vested interest in supporting some particular idea, and may therefore not be impartial, objective judges as to whether such an idea is true. The first definition my computer gives me for “vested interest” is “a personal stake or involvement in an undertaking or state of affairs, especially one with an expectation of financial gain.” There are innumerable reasons why someone may have a vested interest in supporting some idea, some obvious and some not.

Here are some examples when the experts had a vested interest in supporting some particular claim. A cardiologist may recommend a CT scan when he works for a practice that owns some expensive CT scanning machine, and profits in proportion to how often that machine is used. Such an expert opinion is tainted and not trustworthy. In early 2003 we had many military and ex-military figures claiming that Saddam Hussein had weapons of mass destruction. Such persons had a vested interest in going along with the rush to war (in which no such weapons were discovered). If such experts had stated the opposite opinion, saying that the US president was wrong in suggesting that Iraq had such weapons, such experts would have been ostracized within their organizations and social groups, facing various types of penalties, given the “war fever” climate that existed at that time.

Similarly, someone seeking a position as a university professor has a vested interest in going along with whatever doctrines are currently dominant in whatever field he is studying. For example, if you are doing graduate study in neurology or evolutionary biology, you will be much more more likely to be appointed a professor in such fields if you “tow the party line” rather than taking a maverick position in opposition to most of your colleagues. In the same vein, if you are an expert in Catholic theology, you have a vested interest in supporting traditional doctrines rather than opposing them. There aren't many jobs for heretical professors of Catholic theology.

Another reason why a consensus of experts means little is that the pool of experts is not randomly created, and is subject to extremely strong sociological effects that may lead it to poor judgments. In this respect it is interesting to contrast the operations of a jury and the operations of a group of academic experts.

A jury is produced from a random selection of people, which helps to protect it from prejudices held by only a minority of people. If we pick 12 random New Yorkers to judge a murder suspect, we are unlikely to get mostly people with some bias such as “people of that race are born killers.” But there is no such randomness in the formation of the pool of people who end up becoming experts in some topic. It may be that 90% of the people who choose to study neurology are people who previously tended to have thought biases now found in the community of neurologists; and 90% of the people who choose to study evolutionary biology are people who previously tended to have thought biases now found in the community of evolutionary biologists; and 90% of the people who choose to study Biblical theology are people who previously tended to have thought biases now found in the community of Biblical theologians. The result may be some group that ends up being as biased as the group we might have if we selected juries with “Help Wanted” ads like the one below.

Also, juries are exposed to both sides of a case. The prosecution makes its case, and each of its witnesses is cross-examined by the defense attorney. Then the defense makes its case. Both sides make a closing argument. But no such even-handed approach is taken when we are training people to become experts in some particular field. For example, if you study neurology, evolutionary biology, or some type of theology, you are likely to be exposed almost exclusively to those teaching the predominant assumptions of some particular field, with very little exposure to contrasting viewpoints. Once you become an expert in such a field, you are likely to be a kind of “creature of the herd,” a collectivist “organization man” who has been indoctrinated in whatever assumptions have become the sociological norms in some particular subculture. Given such a situation, we should not expect a consensus of opinion within some group of experts to be a very reliable indicator of truth.

CNN once had a fascinating show on how people become members of a biker gang. The gang didn't just quickly accept new members requesting membership. It required that people first serve for years as menial helpers, basically doing any favor that one of the gang members wanted. It is easy to understand the rationale of such a policy. Given such a high cost of admission (in labor and time), it is far more likely that a gang member will conform to the beliefs and behavior of the group, rather than risking expulsion or group condemnation by defying its norms.

A similar situation occurs in regard to becoming a member of many types of elite expert priesthoods. To become, say, a neurologist, you might have to spend $80,000 on graduate school, plus years of study. Having spent that money and all that time, will our new neurologist challenge the accepted assumptions of the group, or will he fall in line, run with the herd, and conform to the norms within the little subculture he has worked so hard (and spent so much) to get into? He will almost certainly do the second thing. This is another reason why a consensus of experts within a field is not something we should be too impressed by. There are often extremely strong sociological factors that may cause herd effects within a group, so a consensus of experts may be no more impressive than the fact that most of a buffalo herd is running in the same direction.

Let's imagine a hypothetical example. Let's suppose there is something called the central doctrine of quonkology, which is advanced by some experts called quonkologists. It might be that 95% of the educated public that has read something about this doctrine consider it to be false. If we were to select the next generation of quonkogists randomly from the public, it would probably be that this central doctrine of quonkology would die. But instead, the next generation of quonkologists will be that small sliver of the population which had a previous tendency to support the central doctrine of quonkology before they signed up to study quonkology, possibly because they shared the intellectual biases and worldview of quonkologists. So can we assume from the favorable consensus of quonkology experts that the central doctrine of quonkology is true? Certainly not.

Whether they be secular or religious, collegiate or non-collegiate, the schools that train experts are often bias magnets. Each type of expert training school attracts people with some particular set of intellectual and ideological biases. The people emerging from such schools may have far more of a particular intellectual bias than the average public. This may lead to very high levels of some intellectual bias within each particular pool of experts, which may help to make its collective judgment unreliable. Once a person signs up for the long process of training to be some type of expert, he may find that the training (and the resulting insular community he becomes part of) act as a bias amplification mechanism. We should hardly be surprised that such “bias-amplified” experts may have a consensus of opinion that is way off the mark. 
 

If a consensus of experts is not a good basis for believing something, what is? Good solid evidence. When you have good solid evidence for something, there's no need to appeal to the fact that there's a consensus of experts. For example, we don't hear people saying to believe in electromagnetism because a consensus of physics experts believes in it. There's no need for that, since it's much more convincing to explain why neither your body nor your smartphone would work if electromagnetism didn't exist. When we hear people appealing to a consensus of experts as the reason you should believe some idea, it's often the case that the evidence for the idea is weak.