The study used a technique called voxel-based morphometry, which attempts to judge brain volume purely by brain scans. The wikipedia.org article on this technique notes some serious reasons for concerns about its reliability. It states the following:
"However, VBM [voxel-based morphometry] can be sensitive to various artifacts, which include misalignment of brain structures, misclassification of tissue types, differences in folding patterns and in cortical thickness. All these may confound the statistical analysis and either decrease the sensitivity to true volumetric effects, or increase the chance of false positives. For the cerebral cortex, it has been shown that volume differences identified with VBM may reflect mostly differences in surface area of the cortex, than in cortical thickness."
The study was one looking at large areas of the brain, looking for differences. This means that the authors had complete freedom to check hundreds of tiny regions of the brain, looking for any differences between their 21 pandits who memorized scriptures, and a group of controls. The problem with this is that a scientist may simply find deviations that we would expect to exist by chance in tiny little brain regions, and then cite these as evidence of a brain effect of memorization. The more regions that are checked, the more likely that some correlation will be found that is purely a chance variation. Similarly, if you were to compare 100 liver regions of 21 Sanskrit scholars with 100 liver regions of 21 ordinary people, you might find 10 or 20 regions of slightly higher tissue density in the Sanskrit scholars. But that would in no way suggest that memories are stored in livers.
Note that the scientist did not claim that the brains of the pandits who memorized scriptures had 10 percent more grey matter than ordinary people. He merely claimed that in "numerous regions" there was a 10% greater difference. If I take 20 random people, scan their brains, and compare them to 20 other random people whose brains I scanned, I will (purely by chance) probably be able to find quite a few little regions in which the first group had more grey matter than the second (as well as quite a few regions in which the first group had less grey matter).
In fact, different regions of random brains typically have a grey matter density that differs by up to 20%. The graph below (link) shows how much grey matter volume varies among a large set of individuals. We see grey matter volume variations of about 20% for ages around 21, and the average age of the 21 Sanskrit pandits was 21.7.
So it is quite unremarkable that a search (among many different tiny brain regions) for differences in grey matter volume of 21 subjects would find many tiny regions where there was a difference of about 10% or 12% or even 15%. Given such large variation in grey matter volumes from subject to subject, we would expect such differences by chance, even if memorization has no effect on grey matter volume. We would expect to find such 10% or 12% differences in grey matter volume when studying random brain regions of any two random groups of 21 subjects (for example, 21 policeman and 21 firemen or 21 baseball pitchers and 21 baseball hitters). It was therefore not-very-honest carnival-barker hype for Hartzell to describe such completely typical and unremarkable variations in grey matter volumes as brain regions that were "dramatically larger."
In the paper, we read that these pandits who memorized scriptures "showed less GM [grey matter] than controls in a large cluster (62% of subcortical template GM) encompassing the more anterior portions of the hippocampus bilaterally and bilateral regions of the amygdala, caudate, nucleus accumbens, putamen and, thalamus." So the study found that in some regions of their brains, these pandits who memorized Sanskrit scriptures had less grey matter than ordinary people, and that in other regions of their brains, they had more grey matter. That is basically what we would expect to find by chance, and provides no good evidence for anything.
Analyzing any brain scan involves a large number of steps, with a large amount of subjective interpretation. An absolute essential for a study like this is a blinding protocol, under which scientists analyzing the brain scans do not know whether the subjects are the ordinary control subjects or the Sanskrit memorization pandits being studied. Such a blinding protocol would be necessary to avoid bias under which scientists looking for some difference would be more likely to find it in a group thought to have such a difference. But the "Sanskrit effect" paper makes no mention of such a blinding protocol being used. I search for the word "blind" in the text, and find no relevant use of it.
On page 23 a technical paper tells us how many subjects we would need to have when doing this kind of "whole brain" study using brain scanning:
Note that the scientist did not claim that the brains of the pandits who memorized scriptures had 10 percent more grey matter than ordinary people. He merely claimed that in "numerous regions" there was a 10% greater difference. If I take 20 random people, scan their brains, and compare them to 20 other random people whose brains I scanned, I will (purely by chance) probably be able to find quite a few little regions in which the first group had more grey matter than the second (as well as quite a few regions in which the first group had less grey matter).
In fact, different regions of random brains typically have a grey matter density that differs by up to 20%. The graph below (link) shows how much grey matter volume varies among a large set of individuals. We see grey matter volume variations of about 20% for ages around 21, and the average age of the 21 Sanskrit pandits was 21.7.
So it is quite unremarkable that a search (among many different tiny brain regions) for differences in grey matter volume of 21 subjects would find many tiny regions where there was a difference of about 10% or 12% or even 15%. Given such large variation in grey matter volumes from subject to subject, we would expect such differences by chance, even if memorization has no effect on grey matter volume. We would expect to find such 10% or 12% differences in grey matter volume when studying random brain regions of any two random groups of 21 subjects (for example, 21 policeman and 21 firemen or 21 baseball pitchers and 21 baseball hitters). It was therefore not-very-honest carnival-barker hype for Hartzell to describe such completely typical and unremarkable variations in grey matter volumes as brain regions that were "dramatically larger."
In the paper, we read that these pandits who memorized scriptures "showed less GM [grey matter] than controls in a large cluster (62% of subcortical template GM) encompassing the more anterior portions of the hippocampus bilaterally and bilateral regions of the amygdala, caudate, nucleus accumbens, putamen and, thalamus." So the study found that in some regions of their brains, these pandits who memorized Sanskrit scriptures had less grey matter than ordinary people, and that in other regions of their brains, they had more grey matter. That is basically what we would expect to find by chance, and provides no good evidence for anything.
Analyzing any brain scan involves a large number of steps, with a large amount of subjective interpretation. An absolute essential for a study like this is a blinding protocol, under which scientists analyzing the brain scans do not know whether the subjects are the ordinary control subjects or the Sanskrit memorization pandits being studied. Such a blinding protocol would be necessary to avoid bias under which scientists looking for some difference would be more likely to find it in a group thought to have such a difference. But the "Sanskrit effect" paper makes no mention of such a blinding protocol being used. I search for the word "blind" in the text, and find no relevant use of it.
On page 23 a technical paper tells us how many subjects we would need to have when doing this kind of "whole brain" study using brain scanning:
"With a plausible population correlation of 0.5, a 1000-voxel whole-brain analysis would require 83 subjects to achieve 80% power. A sample size of 83 is five times greater than the average used in the studies we surveyed: collecting this much data in an fMRI experiment is an enormous expense that is not attempted by any except a few major collaborative networks."
How many subjects did the study of Sanskrit pandits use (a study that refers to its use of "whole brain VBM analysis")? Only 21. This means that it used only a small fraction of the subjects needed for a moderately convincing result, using the approach used. For comparison, a recent paper described two brain scan studies that looked for a link between grey matter volume and psychopathy. The number of subjects who had brain scans were 80 and 64. Clearly our "Sanskrit effect" study has used a number of subjects way too small to support any claims of having found a reliable result.
For all these reasons, the results are not robust evidence for anything. Imagine if you did a study trying to prove the hypothesis that people wearing dark shirts tend to be taller than people wearing light shirts. If you took many photos in school class rooms, and then reported that in "many classroom rows the students wearing dark shirts were taller than average," this would be totally unconvincing evidence for such a hypothesis. We would expect exactly such a thing to be observed by chance even if the hypothesis was not correct. Such a goofy study would be comparable to what Hartzell has reported. He has simply reported grey matter volume variations from place to place such as we would expect to exist by chance in brains where grey matter volumes vary by 20% from person to person, even if memorization has no effect on grey matter volume.
Very strangely, the Hartzell paper shows 14 bar graphs of the density of gray matter volumes in different parts of the brain, showing the difference between Sankskrit memorization pandits and controls; and in 13 out of 14 of these bar graphs, the Sankskrit memorization pandits are shown as having less grey matter volume than the controls. So in general, the bar graphs of his paper contradict the insinuations of Hartzell in Scientific American.
For all these reasons, the results are not robust evidence for anything. Imagine if you did a study trying to prove the hypothesis that people wearing dark shirts tend to be taller than people wearing light shirts. If you took many photos in school class rooms, and then reported that in "many classroom rows the students wearing dark shirts were taller than average," this would be totally unconvincing evidence for such a hypothesis. We would expect exactly such a thing to be observed by chance even if the hypothesis was not correct. Such a goofy study would be comparable to what Hartzell has reported. He has simply reported grey matter volume variations from place to place such as we would expect to exist by chance in brains where grey matter volumes vary by 20% from person to person, even if memorization has no effect on grey matter volume.
Very strangely, the Hartzell paper shows 14 bar graphs of the density of gray matter volumes in different parts of the brain, showing the difference between Sankskrit memorization pandits and controls; and in 13 out of 14 of these bar graphs, the Sankskrit memorization pandits are shown as having less grey matter volume than the controls. So in general, the bar graphs of his paper contradict the insinuations of Hartzell in Scientific American.
Dear Mark and fellow readers
ReplyDeleteI don't normally respond to opinion pieces about the research we published in Neuroimage (https://www.ncbi.nlm.nih.gov/pubmed/26188261--see this link in my Academia.edu page for full download of both the paper and the Supplementary Materials: https://www.academia.edu/14172671/Brains_of_verbal_memory_specialists_show_anatomical_differences_in_language_memory_and_visual_systems) since many such pieces are filled with personal opinions of the writers that have little relevance to the research. In my view Mr. Mahin's piece is one such, though it is filled (a bit unusually?) with outright false claims and factual errors. First the Scientifica American blog piece based on the Neuroimage paper was published back in January 2018, not this week of September 2019 as Mahin claims. Second, the bar graphs Mahin refers to, as is readily seen in the Neuroimage paper, refer to different subcortical structures where, indeed, whole brain GM analysis showed less GM in Pandits than controls. More specific ROI analysis of the hippocampi showed substantial GM increases in the right HF, and part of the left HF. We did not conduct an ROI-specific analysis of the GM of the other subcortical structures in this study. As stated clearly in the Neuroimage paper, "The hippocampal-optimized VBM procedure indicated a large portion of the posterior-middle right HF where Pandits had greater GM than controls (see Fig. 4, and see Supplementary Table 7 for cluster specifics). The volume of this region formed between 73 and 98% of the hippocampal mask." This hippocampal difference finding was confirmed with two other methods (reported in the Supplementary Materials). Third, the Cerebellum showed ~33% increased GM in Pandits relative to controls, a fact Mahin neglects to note. Fourth, the ~10% relative increase in GM in the neocortex in the Pandits relative to controls is not in "tiny" brain regions as Mahin suggests but rather in large clusters -- the largest being 22,546 voxels (hardly a tiny region), as fully reported in the tables in the Supplementary Materials. Mr. Mahin evidently also misread the SciAm piece, since he claims: "The story author has given this alleged effect the catchy name of "the Sanskrit effect." But when we take a close look at the study, co-authored by Hartzell, we find no robust evidence for any brain change caused by memorization." In fact neither the Neuroimage piece nor the SciAm piece make the claim that the differences we saw between the professional Pandits and the controls was cause by memorization--a cross-sectional study of the type we conducted cannot prove that one way or the other--rather it would require a longditudinal study that has yet to be conducted.
Readers can draw their own conclusions from the paper and the SciAm piece.
Thanks for your reply, anonymous replier. You have not done anything to substantiate your reply that my post was "filled...with false claims and factual errors." The only factual error you have found was my original unimportant claim that the Scientific American piece was published this week. That was an easy error to make, because this week on the widely-read Psychology reddit page, there was a link to the Scientific American article, as if it was recent news. Now, as for your claim that "the Cerebellum showed ~33% increased GM in Pandits relative to controls, a fact Mahin neglects to note," that is an extremely misleading statement. You thereby imply that in general the cerebellum of the Sanskrit pandits had 33% more grey matter. But the study found no such thing (which would have been a huge discovery if it had been found). Instead, what the study found was that in 33% of the cerebellum, there was more grey matter in the Sanskrit pandits. That is an unremarkable finding that we would expect to get by chance. We might expect, for example that 33% of the cerebellums of the pandits would have more grey matter than some controls, and that 33% of their cerebellums would have less grey matter than some controls. As for your claim that "neither the Neuroimage piece nor the SciAm piece make the claim that the differences we saw between the professional Pandits and the controls was cause by memorization," the Scientific American very much does insinuate such a claim. In fact, the headline of the Scientific American article (just underneath the title) says, "MRI scans show that memorizing ancient mantras increases the size of brain regions associated with cognitive function." As for your complaint about my reference to "tiny regions," I made no claim that the differences you found were only in "tiny regions." I merely used the phrase "tiny regions" in reference to what would be expected to be found by chance.
ReplyDeleteThe exact statement in the scientific paper (relating to grey matter) was "In the cerebellum Pandits showed greater GM than controls in multiple bilateral structures (Fig. 2A), encompassing 34% of the total GM in the cerebellar template." Now that very clearly is merely a statement that in various areas totaling 34% of the cerebellum, there was more grey matter -- and not at all "33% increased GM in pandits." The pandits didn't have 33% more grey matter in the cerebellum. It was merely found in 34% of the regions checked there was more grey matter, which is what we would expect by chance (and also that 33% of the regions checked would have less grey matter).
DeleteAs for your complaint that the 22546 voxel region (the largest area reported with a grey matter difference) was "hardly a tiny region," it is been estimated here that a full high-resolution brain scan is about one million voxels.
Deletehttps://www.quora.com/Connectome-How-many-voxels-are-in-the-Human-Brain
So the 22,546 voxel area is about one fiftieth of the brain, or 2% of the volume of the brain. There would be no clear error involved if you referred to an area so small as either "small," "very small," or "tiny" (since such adjectives are ambiguous).