Brain Images, Babies, and Bathwater: Critiquing Critiques of Functional Neuroimaging PDF

Brain Images, Babies, and Bathwater: Critiquing Critiques of Functional Neuroimaging By Ma rtha J. Fa ra h S ince the mid-1980s, psychologists and neurosci- of imaging’s limitations without “...

Brain Images, Babies, and Bathwater: Critiquing Critiques of Functional Neuroimaging By Ma rtha J. Fa ra h S ince the mid-1980s, psychologists and neurosci- of imaging’s limitations without “throwing the baby out entists have used brain imaging to test hypotheses with the bathwater.” about human thought processes and their neural The criticisms I review here can be grouped into four instantiation.1 In just three decades, functional neuroim- general categories, although there is a degree of overlap aging has been transformed from a crude clinical tool among them and many have been combined in the lit- to a widely used research method for understanding the erature despite their conceptual distinctness. The first human brain and mind. Such rapidly achieved success is broad category concerns the nonidentity of, or the gap bound to evoke skepticism. A degree of skepticism to- between, the neural events being studied and the images ward new methods and ideas is both inevitable and useful that purportedly represent them. in any field. It is especially valuable in a science as young as cognitive neuroscience and its even younger siblings, Inferential Distance and the Objects of Imaging social and affective neuroscience. Healthy skepticism encourages us to check our assumptions, recognize the limitations of our methods, and proceed thoughtfully. Skepticism itself, however, also must be examined. A s Adina Roskies has pointed out, functional brain images are easily misunderstood as photographs of brain function.2 There are, however, numerous types of Functional neuroimaging has attracted a substantial mismatch in the relationship between published func- amount of skepticism from inside and outside the fields tional brain images and the brain activity they represent. of psychology and neuroscience. In this article, I review Two of these mismatches form the basis of many of the the most commonly voiced criticisms of functional neu- common criticisms reviewed here. roimaging. In the spirit of healthy skepticism, I will criti- Blood versus brain. The signal measured in function- cally examine these criticisms themselves. Each contains al magnetic resonance imaging, as explained by Geoffrey at least a kernel of truth, although I will argue that in Aguirre in his essay in this collection, is a characteris- some cases the kernel has been overextended in ways tic of blood rather than brain tissue. The dependence that are inaccurate or misleading. In other cases, the on hemodynamic proxies for brain activity strikes some criticisms are valid as presented and deserve the careful commentators as a fundamental flaw in the functional attention of imaging researchers. The goals of this article neuroimaging enterprise. I offer three examples of this are to distinguish between reasonable and unreasonable critique, from philosophers, a science writer, and a psy- criticisms and to identify some general characteristics of chologist. the two categories. In this way I hope to encourage the Philosophers Valerie Hardcastle and Matthew Stewart fair and realistic evaluation of functional neuroimaging question “the excited hoopla over fMRI and other im- as a scientific method and to foster an understanding aging techniques” by pointing out that fMRI informs us about activity only in a relatively large area of brain Martha J. Farah, “Brain Images, Babies, and Bathwater: Critiquing tissue (on the order of millimeters) and can inform us Critiques of Functional Neuroimaging,” Interpreting Neuroimages: An Introduction to the Technology and Its Limits, special report, Hastings Center only at relatively long time intervals (on the order of sec- Report 45, no. 2 (2014): S19-S30. DOI: 10.1002/hast.295 onds).3 They suggest that this poor spatial and temporal SPECIAL REP ORT: I nt er p r et in g N eur o im a ges: An I n trod u c ti on to th e Te c h n ol og y a n d I ts L i mi ts S19  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License ably with neural activity. Indeed, despite con- Figure 1. siderable research on the subject, we do not Average global temperature, 1880-2004 yet know how fMRI corresponds to specific (Source: Goddard Institute for Space Studies) aspects of neural activity, be they the summed synaptic activity of small populations of neu- rons, spiking of neurons, or other physiologi- cal phenomena.6 But let us put these criticisms in perspec- tive. The concern that fMRI shows us blood oxygenation rather than neural activity direct- ly should be weighed alongside the fact that little of what we call science involves direct observations of the subject matter of interest. Cosmologists make inferences about earlier states of the universe by measuring background microwave radiation. Chemists determine the composition of samples by heating or shining light on them and measuring emission or ab- sorption spectra. Climate scientists measure tree rings to study climate trends over previous centuries. Complaints that functional neuro- images do not “show” brain activity appear to be based on a naïve view of science and its methods. Of course, compared to the cos- mologists’ or chemists’ spectroscopy, we have only a weak grasp on the nature of the linkage resolution creates an imprecise representation of neural ac- between fMRI and neural activity. Nevertheless, blood- tivity and write, “This imprecision forecloses the possibility oxygen-level-dependent (BOLD) fMRI detects a relation- of directly connecting single-cell activity—which operates ship between neural activity and oxygenation that is strong three to four orders of magnitude smaller and faster—with enough to make it a useful research tool. larger brain activation patterns. What are we to do? The Nikos Logothetis, an influential voice of caution on the answer given all too often by neuroscientists is to fudge.” interpretation of fMRI data, has pointed out that similar In a popular book on the brain, Judith Horstman like- problems regarding indirectness apply to other neurosci- wise argues that “an fMRI is not a direct image of your brain ence methods: “Electrical measurements of brain activ- showing mental activity. It’s an image created through in- ity, including invasive techniques with single or multiple directly measuring the flow of oxygenated blood and then electrodes, also fall short of affording real answers about correlating that information to something you are doing network activity. Single-unit recordings and firing rates or thinking at the time, and the blood flow response takes are better suited to the study of cellular properties than of time. So it’s a stretch to say that there is cause and effect neuronal assemblies, and field potentials share much of the and to relate this blood surge to a specific activity.”4 ambiguity discussed in the context of the fMRI sign.” He Perhaps the strongest statement of concern over the also affirms that “despite its shortcomings, fMRI is cur- nonequivalence of blood flow and brain activity can be rently the best tool we have for gaining insights into brain found in the writings of psychologist William Uttal, a vo- function.”7 cal critic of functional neuroimaging research, who recently In sum, it is not the case that inferences based on func- wrote, “fMRI is as distant as the galvanic skin response or tional brain imaging are, in the words of the critics quoted pulse rate from cognitive processes.”5 earlier, “fudges” or “stretches.” There is indeed a “cause and The undeniable truth in these criticisms is that blood effect” relationship between neural activity and blood oxy- flow and oxygenation are not the same as brain activity. genation levels, even if the nature of that relation is imper- Blood flow and oxygenation occur at different spatial scales fectly understood. and over different time courses than brain activity. More Functional brain images as fabrications. Another way fundamentally, even if we allow for those spatial and tem- in which the relation between functional brain images and poral limitations, there is no guarantee that cerebral blood their objects has been questioned concerns the importance flow or blood oxygenation will correlate precisely or invari- of the decisions that researchers must make (described by S20 March-April 2014/ H AS TI N GS CE NTE R RE P O RT  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License The concern that fMRI shows blood oxygenation, not neural activity directly, should be weighed alongside the fact that little of what we call science involves direct observations of the given subject matter. Aguirre in his essay) regarding what aspects of brain activity tures—for example, the range from the lowest to highest to represent and how to represent the selected activity. The naturally occurring temperatures. Doing so would put the worry, in this case, is that the images are more researcher data into a different context, which might be useful for inventions than researcher observations. some purposes, but for most purposes it would be coun- For example, scientists and nonscientists alike have terproductive; it would simply make the relationships of regarded the use of color-coding with suspicion. Writer interest hard to see. Similarly, the color scale used for func- Michael Shermer states that “[c]olors exaggerate the effects tional brain images spans a small range of BOLD signal in the brain.... The coloring is artificial, and the process of change in order to maximize the visibility of the distribu- coloring the regions is even more misleading.”8 Horstman tion of relevant activations. echoes these criticisms, warning that despite significant Could one fabricate results, in the sense of changing differences in the colors used, “differences in activity lev- the pattern of activation, by changing color scales? It is not els are tiny.”9 In an article entitled “Some Cautions about possible to boost or minimize the activation of one region Jumping on the Brain-Scan Bandwagon,” psychologist and by changing scales without boosting or minimizing other textbook author Carole Wade warns that “decisions about equally activated regions. Choices of color scale or contrast color scales... can accentuate or minimize the contrasts affect the salience of certain features, but this is equally among different brains or brain areas. Such decisions can true of other biological images, including micrographs, affect whether the gorgeous images we see at conferences, gels, and Western blots.11 The issues here are not unique to in articles and textbooks, and in the popular press will be brain imaging, nor do they indicate that colored images are striking, ho-hum—or even misleading.”10 inherently or even typically deceptive. The element of truth in this criticism is that color-cod- ing is arbitrary. The choice of which color represents which Metatheoretical Assumptions and Goals of numerical value is dictated by both convention and conve- Neuroimaging nience. Conventionally, warmer and brighter colors repre- sent higher activation. But is this in some way prejudicial? No more so than plotting numerical data on axes where higher numbers appear higher on the page. For functional S cientific theories and methods inevitably influence one another. In the case of functional brain imaging, it has been suggested that the method seems particularly com- images or for the coordinates on a traditional Cartesian patible with certain assumptions about the mind-brain graph, one could reverse the scale to remind viewers of the relation and encourages certain types of theories while pre- conventional nature of these data representations, but the venting others from being tested. Specifically, functional value of doing so seems small compared to the inconve- neuroimaging has been criticized for encouraging research nience to readers. aimed merely at localizing psychological functions, for be- A related criticism concerns the calibration of color ing incapable of testing psychological theories, for assum- scales in images. The criticism is that large differences in ing a modular relation between mental and neural systems, color can mislead the naïve viewer into thinking that the and even for assuming a one-to-one correlation between differences in neural activity are also large, whereas, to use these systems. Horstman’s word, they are in fact “tiny.” But do these ef- Localization versus explanation. Neuroimaging has forts at making different levels of brain activation visible been caricatured as a form of phrenology, with the research make the images, again using Horstman’s word, “mislead- goal being simply to associate a psychological function with ing”? Not at all. Again, the analogy with Cartesian graphs a specific part of the brain. It is easy to see how this mis- can help. Consider the graph of average global tempera- understanding could occur, given the archetypal functional tures shown in figure 1. The calibration of the Y-axis spans brain image bearing color splotches affixed to various parts less than two degrees Celsius because this makes the rel- of a structural MRI. evant relationships among data points salient and enables The philosopher Jerry Fodor has suggested that the readers to glean an accurate sense of both the variability question of where in the brain a given psychological ac- and the trend of the data points. These data could have tivity occurs is scientifically trivial. He writes, “It isn’t, af- been plotted on a graph showing a fuller range of tempera- ter all, seriously in doubt that talking (or riding a bicycle SPECIAL REP ORT: I nt er p r et in g N eur o im a ges: An I n trod u c ti on to th e Te c h n ol og y a n d I ts L i mi ts S21  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License or building a bridge) depends on things that go on in the choosing and arranging the order of stimuli and then mea- brain somewhere or other. If the mind happens in space at suring the effects of preceding one stimulus with another, all, it happens somewhere north of the neck. What exactly it is possible to determine the proportion of neurons that turns on knowing how far north?”12 Elsewhere in the same the two stimuli activate in common, in effect, the similar- essay he explains why our alleged obsession with localiza- ity of neural coding between the stimuli. When one knows tion is not just silly but wasteful: “Science is expensive.... the nature of representational similarity, one knows much If you put your money (which is to say, our money) into about the nature of representations.17 Using this approach, the elaborate technology required to establish localizations researchers have been able to distinguish between view- of mental functions by imaging techniques, you almost cer- point-dependent representations of an environmental loca- tainly take it out of other kinds of psychological research. tion (akin to a photograph taken from a particular vantage Likewise in respect of the time and money that is required point) and viewpoint-invariant representations (similar to to train people to do science; graduate students, too, are a a map representation of location).18 limited resource.”13 Other decidedly nonphrenological methods of analysis Uttal, whose 2001 book-length critique of neuroimag- take advantage of neuroimaging’s ability to give informa- ing was titled The New Phrenology, suggests that the quest tion about the state of the entire brain, which contrasts to localize psychological functions in the brain is a primary, with the necessarily piecewise approach of nonimaging and misguided, goal of neuroimaging research. He notes methods such as single-cell recording and lesion methods. that “a considerable amount of PET [positron emission Analyses of functional connectivity reveal which subset tomography] and fMRI localization has simply confirmed of areas show correlated activity, suggesting that they are some things that we have long known,”14 and he contrasts working together.19 Such analyses pick out functional net- such research with research that actually tries to discover works of areas, which may change depending on the task “how the brain computes, represents, encodes, or instanti- conditions, and hence put strong constraints on the nature ates psychological processes.”15 of the organization of cognitive subsystems. It is hard to disagree with Fodor and Uttal’s observa- Localization is also generally beside the point in multi- tion that localization is, in itself, a questionable scientific voxel pattern analysis methods, described elsewhere in this goal. However, most contemporary neuroimaging research report by Aguirre. Indeed, the results of such studies can has other goals. In the early years of PET and fMRI, each often be stated without any reference to anatomical local- method was used to image processes whose brain localiza- ization. These methods enable researchers to detect specific tions were already well known on the basis of lesion stud- mental states and decode how specific stimuli or events ies or single-cell recording in animals. This exercise helped are represented in the brain. None of the foregoing uses of to validate the new methods, testing them in order to see neuroimaging fits the description of “neophrenology.” whether they produced the expected localizations. Once Relevance to psychological theory. While anatomical lo- these localizations were confirmed, confidence in the abil- cation is an intrinsic property of neuroimaging data, the ity of functional imaging to detect regional brain activity use of these data is not confined to the study of localiza- increased. Researchers then began to use the method to in- tion any more than the use of reaction time, the primary vestigate cases in which it was not known which brain areas methodology of cognitive psychology, is confined to the were likely to be recruited for a given psychological func- study of cognitive speed. Both localized activations and tion, as well as to address questions that were not primarily response latencies are used by researchers to test psycho- questions of localization. Of course, studies of localization logical hypotheses. Some critics acknowledge this use of for localization’s sake are still occasionally published, just as functional imaging but question whether it has been effec- purely descriptive studies with other methods can be found tive as a means of testing psychological theories.20 As with in any field of science, but the bulk of functional neuroim- the preceding objections, there is validity to this one. Let aging research in the twenty-first century is not motivated us consider the sense in which it is valid as well as the sense by localization per se. in which it misses the mark. Further reason to reject the “neophrenology” charge In response to examples of how neuroimaging can con- comes from neuroimaging methods that exploit other fea- firm or disconfirm psychological theories, Max Coltheart tures of activation in addition to location. For example, has provided alternative explanations of specific findings “adaptation paradigms” make use of changes in activ- to show that they are not, in fact, decisive.21 What gets lost ity during the course of perception or cognition, specifi- in the debate is the fact that decisive experiments are not cally the diminution in response to a repeated stimulus or generally possible in psychology, and it would be equally operation when the same set of neurons is reactivated.16 difficult to offer examples of traditional cognitive psychol- This is because neurons “adapt” to stimulation, reacting ogy’s success by this criterion.22 The phenomena under less strongly if they have recently been active. By carefully study within cognitive psychology and neuroscience are S22 March-April 2014/ H AS TI N GS CE NTE R RE P O RT  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License Can functional brain imaging contribute to confirming psychological hypotheses in roughly the way behavioral studies do? too complex, and the associated theories have too many The kernel of truth here is that early approaches to the degrees of freedom, to hope for decisive single findings. design and analysis of functional neuroimaging experi- A fairer and more realistic question is this: Can func- ments were best suited to studying relatively small numbers tional brain imaging contribute to confirming psychologi- of macroscopic regions of activation, ignoring questions of cal hypotheses in roughly the way behavioral studies do? representation within areas or complex interactions among Can functional imaging, experiment by experiment, rule areas. The method of subtraction (again, see Aguirre), out the more straightforward alternative hypotheses and which was first used to show how functional brain imaging leave progressively more complex or strained alternatives can illuminate cognitive processes, assumes that a cognitive in relation to the supported hypothesis? Many fruitful re- process A will have the same neural instantiation whether search programs answer this question in the affirmative. it is accompanied by cognitive process B, C, D, or E.30 In Different examples of issues in cognitive psychology that this way, the subtraction method assumes fixed, context- imaging has helped to resolve have been offered.23 My own independent modules. Of course, analytic approaches such favorite example concerns the relation between mental im- as adaptation designs and network modeling, discussed agery and perception, a long-standing issue in cognitive earlier, show how fMRI can transcend this limitation. psychology.24 Does the visual system do “double duty” for As Erik Parens pointed out in the course of the meetings perceptual processes and mental images generated from that gave rise to this collection of essays, the history of neu- memory, or is the system used to generate mental images roimaging may be not unlike that of behavioral genetics in from memory independent of the visual system? Measures this regard. In the 1980s and ’90s, encouraged by the dis- of localized brain activity as rudimentary as electroenceph- covery of several examples of rare medical diseases that were alogram (EEG) and single-photon emission computed caused by single genes, some researchers set out to discover tomography (SPECT),25 later confirmed with PET and single genes responsible for common psychiatric illnesses fMRI, ruled out a set of alternative explanations that had such as depression, schizophrenia, and autism. In effect, plagued the behavioral approach to this question, with the these behavioral geneticists were also assuming a simple evidence indicating at least some shared representations.26 and modular theoretical framework, with individual genes Of course, the original question was replaced by new ques- responsible for the psychological phenomena of interest. tions, inquiring, for example, more specifically as to which With time and the accumulation of research results show- visual representations are shared with imagery, but this was ing the inadequacy of this assumption, behavioral genetics progress. Recently, Mara Mather and colleagues surveyed moved to more complex models, in which genes exert their a variety of research programs and identified four distinct effects on behavior through complex networks of interac- ways in which localized brain activity can be used to test tion with one another and with the environment. It wasn’t cognitive theories.27 the case that the gene as a unit of analysis was useless, but Biasing hypothesis generation. It has also been suggest- that genes had to be seen in still more complex terms. ed that the use of imaging constrains the kinds of theories Finally, the concern that brain imaging puts scientific of mind-brain relations that will be devised and tested. The blinders on researchers, allowing them to see only simple concern is not that functional brain imaging is merely an modular systems, is assuaged by the concurrent use of other exercise in localization or that it is incapable of testing psy- methods. Hypotheses to be tested are selected in part based chological hypotheses. Rather, the concern is that it directs on research with other methods of neuroscience and psy- us to think about brain function in the wrong way. First, chology, and results are interpreted in part based on re- it invites us to focus on a subset of the relevant data. The search with these other methods.31 problem, writes Uttal, is “the mistaken idea that when all Wanton reverse inference. If there were a one-to-one lesser peaks are reduced to invisibility by arbitrary scaling, relationship between brain regions and psychological pro- the largest remaining peak represents the sole locale of a cesses, as assumed in strict modular hypotheses of brain particular cognitive process.” Second, imaging invites us function, then it would be easy to infer what psychological to frame hypotheses “at the wrong (macroscopic) level of processes are under way at any given moment simply by analysis rather than the (correct) microscopic level.”28 The observing which brain regions are active. Given that the result is “misdirected attention and effort”: “we are doing brain-mind relation is far more complex than that, with what we can do when we cannot do what we should do.”29 a single psychological process of interest typically engag- ing multiple regions and a single region typically involved SPECIAL REP ORT: I nt er p r et in g N eur o im a ges: An I n trod u c ti on to th e Te c h n ol og y a n d I ts L i mi ts S23  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License in the implementation of multiple psychological processes, pattern of brain activation when giving a response more inferring a psychological process from an observed brain closely matches that previously associated with true or de- activity is not straightforward. This has not stopped re- ceptive responses.38 In principle, and with the right base of searchers from attempting to make such inferences, ignor- evidence, these reverse inferences could lead to valid con- ing the intrinsic ambiguity of a single region or pattern of clusions along the lines of “there is a 75 percent chance activation in a single experimental context, considered in that the subject” either wants the product, would respond isolation. to Prozac, or is telling a lie. In most cases, however, the This type of inference, going from an observation of empirical groundwork needed for these claims is far from brain activation to an inference about the psychological complete. For example, for all we know now, psychological process that caused it, was called “reverse inference” by processes other than lying could produce the “lying pat- Russell Poldrack. The name highlights a difference be- tern” with high probability.39 tween this research practice and the more common, and The multivoxel pattern analysis (MVPA) technique (see less problematic, “forward inference” practice of manipu- Aguirre) is in effect a combination of reverse and forward lating psychological processes and observing resulting brain inference approaches. Reverse inferences can be made with activation.32 confidence only when one knows the full range of psycho- Although reverse inference has often been criticized, logical processes that could produce a given pattern of ac- it is not, in itself, invalid. If one has done due diligence tivation under the circumstances of the study. In MVPA to ascertain the range of psychological processes that can studies, forward inference is first performed on all of the activate a region under a given set of circumstances, then psychological states that reverse inference will later be used appropriately framed reverse inferences will be possible.33 to select among. When the time comes for reverse infer- Unfortunately, it has often been used wantonly, interpret- ence, it is in effect a very large multiple-choice among the ing a pattern of activation without knowing, or acknowl- psychological states that were the subject of forward infer- edging, the variety of psychological processes that could ence. More open-ended uses of reverse inference, in situa- produce that pattern. A celebrated example of such wan- tions where one does not know all of the subject’s possible ton reverse inference appeared in a 2007 New York Times psychological states, would require more exhaustive pro- op-ed written by neuroscientists Marco Iacoboni, Joshua grams of forward inference research to have been complet- Freedman, and Jonas Kaplan during that year’s presiden- ed and might yield an answer taking the form of “state A tial primaries. They presented their findings from an fMRI with probability X, state B with probability Y, and so on.” study of undecided voters viewing still photos and videos As with many of the other concerns reviewed here, un- of the leading candidates. On the basis of activity in the an- warranted reverse inference is not unique to functional terior cingulate cortex (ACC), known from other studies to neuroimaging. It will affect any technology in which a spe- become active when subjects feel conflicting response ten- cific cause is inferred from the presence of an effect that dencies, the authors concluded that voters felt ambivalence could have resulted from other causes. For example, in the toward Hilary Clinton and were “battling unacknowledged use of conventional polygraphy for lie detection, responses impulses to like” the candidate.34 can be accompanied by autonomic reactions for reasons Several cognitive neuroscientists harshly criticized the other than deception.40 In prostate-specific antigen testing, op-ed, pointing out numerous flaws in the research design elevations of the protein prostate-specific antigen can re- and data analysis35—and emphasizing the wanton use of sult from causes other than a growing tumor.41 In sum, the reverse inference. The problem with the reverse inference problem with reverse inference is not unique to brain im- in this case was that many other psychological states also aging, and reverse inference is not inherently problematic. activate the ACC. Activity in this area has been elicited by The problem is with making a reverse inference without processes as diverse as attention to one’s own heartbeat and the relevant knowledge from forward inference. emotional regulation.36 Many of the commercial applications of functional Neuroimaging’s Slippery Statistics brain imaging involve reverse inference. For example, neu- romarketing relies on the ability to infer liking and want- ing from patterns of brain activation.37 Similarly, the use of functional brain images for diagnosis in some for-profit V irtually all scientific research depends on statistical analysis at some critical juncture or other, but func- tional brain imaging research is particularly dependent on psychiatric clinics is also based on the premise that certain statistics. As described earlier by Aguirre, extensive signal patterns of activation can be used to infer the presence of processing and statistical analysis intervene between the certain disorders or the suitability of certain treatments. data acquired from the scanner and the published image. Brain-based lie detection promises to distinguish true from A number of criticisms of functional brain imaging have deceptive responses by determining whether a person’s focused on the statistics involved. S24 March-April 2014/ H AS TI N GS CE NTE R RE P O RT  1552146x, 2014, s2, Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/hast.295 by Cochrane Mexico, Wiley Online Library on [22/06/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License When carried out properly, statistical analyses deepen our understanding of the data and the larger reality from which they were sampled. Statistical inference versus direct observation. Some necessarily limited measurements. Like every other aspect criticisms concern the sheer amount of statistical analysis of scientific research, statistical analyses can be done well or involved in producing a functional brain image. The use poorly, with objectivity or bias, but the use of statistics per of statistics often involves substituting estimated values for se is not a problem. raw data. When the very earliest stages of image processing The kernel of truth in the criticisms just discussed is consist of replacing measured signal values with estimated that the extensive use of statistics provides equally extensive values, the result can be viewed as a fabrication, with all the opportunities for error. Two such errors are reviewed in the negative connotations of that term discussed earlier. next two sections. In addition, the extensive statistical processing involved Multiple comparisons. Functional brain imaging is sus- in neuroimaging offers many opportunities to distort ceptible to a particular kind of statistical problem that can scientific evidence and therefore makes some critics sus- exaggerate the reliability of findings. The problem arises picious. For example, neuropsychologist Carlo Umilta is because of the enormous number of statistical tests that quoted as questioning, “Would coloured images be so con- can be carried out with image data. A functional MRI may vincing even if readers knew those images are the result of contain 50,000 voxels (see the essay by Aguirre for expla- an elaborate sequence of ‘cleaning actions,’ each one char- nation of “voxel”), and each of those voxels could be the acterized by a not insignificant error probability?”42 More site of an independent statistical test comparing the value extreme skepticism comes from neuroscientist Steven Rose. of the BOLD response measured in that small bit of brain Commenting on Fodor’s critique of imaging, he wrote that between the conditions of the experiment. Statistical tests the “images are marvelously seductive, but by the time yield a “significance level,” which is the probability that you see them they have been so massaged as to risk being the observed difference between two conditions was due thoroughly misleading.”43 Michael Shermer observes that to chance variation alone. Given that experiments are de- “[b]rain images are statistical compilations,” and he advises signed so that the hypothesis of interest predicts a “real” his readers to “keep all [these many stages of data analysis] difference—a difference, that is, that would be expected in mind next time you see one of those colorful brain scans. any time the experiment was run and not just on occasions... [Such images are] highly misleading.”44 Even the wide- when it happens by chance—one should employ a fairly ly employed and basic statistical process of averaging can low, and thus stringent, significance level. The conven- seem problematic in the context of imaging: Carole Wade tional cut-off for considering a finding reliable or “real” is offers that the “vexing problem of individual differences in a probability of less than one in twenty that the difference brain anatomy,” which may make the “uniqueness of fin- between conditions is due to chance, usually written as gerprints or facial features seem simple by comparison,” is “p