The What, When, Where, and How of Visual Word Recognition PDF

Review The what, when, where, and how of visual word recognition Manuel Carreiras1,2, Blair C. Armstrong1, Manuel Perea1,3, and Ram Frost1,4,5 1 Basque Center for Cognition, Brain, and Language, San Sebastian, Spain 2 IKERBASQUE Basque Foundation for Science, San Sebastian, Spain 3 Universita...

Review The what, when, where, and how of visual word recognition Manuel Carreiras1,2, Blair C. Armstrong1, Manuel Perea1,3, and Ram Frost1,4,5 1 Basque Center for Cognition, Brain, and Language, San Sebastian, Spain 2 IKERBASQUE Basque Foundation for Science, San Sebastian, Spain 3 Universitat de València, València, Spain 4 The Hebrew University of Jerusalem, Jerusalem, Israel 5 Haskins Laboratories, New Haven, CT, USA A long-standing debate in reading research is whether information often tells us something important about printed words are perceived in a feedforward manner on ‘what’ types of representations are activated during the basis of orthographic information, with other repre- visual word recognition and ‘how’ readers eventually sentations such as semantics and phonology activated recognize words [1–3]. subsequently, or whether the system is fully interactive However, a comprehensive account of how complex and feedback from these representations shapes early stimuli such as words are processed requires a detailed visual word recognition. We review recent evidence description of the temporal flow of information and eluci- from behavioral, functional magnetic resonance ima- dation of ‘when’ the internal representations of words (e.g., ging, electroencephalography, magnetoencephalogra- letters, syllables, morphemes, lexical entries) are acti- phy, and biologically plausible connectionist modeling vated. Figure 1 presents contrasting frameworks. In this approaches, focusing on how each approach provides respect, ‘when’ questions constrain any theory of ‘how’ by insight into the temporal flow of information in the detailing the sequence of events from stimulus presenta- lexical system. We conclude that, consistent with inter- tion to word recognition. In fact, one of the oldest debates in active accounts, higher-order linguistic representations visual word recognition concerns the demarcation between modulate early orthographic processing. We also dis- bottom-up and top-down processing, asking whether or not cuss how biologically plausible interactive frameworks the visual stimulus feeds into the lexical level in a pre- and coordinated empirical and computational work can dominantly hierarchical manner, wherein orthographic advance theories of visual word recognition and other representations feed into higher-level linguistic represen- domains (e.g., object recognition). tations, or whether higher-level linguistic information such as phonological and morphological structure exerts The what, when, where, and how of visual word a top-down influence on visual orthographic processing recognition relatively early (Box 2). Cognitive neuroscience has A viable theory of visual word recognition needs to specify rekindled this debate through the introduction of techni- ‘what’ the building blocks of a printed word are and ques such as electroencephalography (EEG) and magne- describe ‘how’ they are processed and assembled to give toencephalography (MEG), which have the appropriate rise to word identification. These central ‘what’ and ‘how’ temporal resolution to track the time course of processing. questions have been the focus of research (and contro- Note, however, that the ‘where’, ‘what’, ‘how’, and ‘when’ versy) in cognitive science since its very beginning, and questions are to a large extent interdependent. The human have traditionally been addressed by combining inven- brain is generally constructed so that the trajectory of tive experimental designs and reaction time (RT) mea- increased complexity, in terms of moving from relatively sures (Box 1). More recently, the availability of simple microfeature representations (e.g., the line seg- techniques such as functional magnetic resonance ima- ments in a letter) to complex, higher-order representations ging (fMRI) have provided new opportunities to ask pre- (e.g., a representation of the whole word form) is occipital- cise ‘where’ questions, focusing on locating the to-frontal, whereas the trajectory of high-level modulation neurocircuitry involved in recognizing printed words. is frontal-to-occipital. Because ‘where’ information is cor- Given the architectural constraints of the brain, ‘where’ related with the flow of processing (early/simple or late/ higher-order), locations of brain activations are often taken Corresponding author: Carreiras, M. ([email protected]). Keywords: visual word recognition; visual word form area; orthographic processing; to support claims regarding the temporal order of proces- neural connectivity; computational modeling; feedback versus feedforward informa- sing. Here we discuss the potential danger of using evi- tion. dence of ‘where’ to make inferences about ‘when’ (and 1364-6613/$ – see front matter ‘how’), review the findings obtained using techniques with ß 2013 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.tics.2013.11.005 the appropriate temporal resolution for tracking the time course of printed word processing, and point to desirable cross-fertilization between behavioral data, neuroimaging techniques, and neurobiologically plausible computational 90 Trends in Cognitive Sciences, February 2014, Vol. 18, No. 2  Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 Box 1. Measuring time courses in behavioral research Although behavioral investigations are useful for understanding time necessary for activation of that information (e.g., orthographic, visual word recognition, these studies suffer from an inherent phonological, morphological, or semantic information). Nonetheless, limitation: they only provide an end-state reflection of the state of this procedure has limitations , such as a lack of ecological processing via an indirect behavioral response (e.g., lexical decision validity. A related and more ecologically valid technique is to present time as signaled via a key press). Thus, these techniques do not the words in the context of normal silent reading while the provide direct insight into the internal temporal dynamics underlying participants’ eye movements are registered. Of particular interest ‘how’ different representations are activated. Moreover, these is the very early parafoveal preview benefit effect using the boundary approaches run the risk of producing data that are contaminated by technique, in which the relationship between a parafoveal preview pre- and post-lexical processes (e.g., decision-making). and a target word is manipulated. Specifically, the parafoveal preview These limitations notwithstanding, techniques that provide relevant is replaced by the target word once the fixation crosses an invisible indirect insight into the time course of different processes have been ‘boundary’ located next to the target word. Differences in fixation developed that relate to the ‘when’ question regarding feedforward duration on the target word caused by different structural manipula- and feedback processes. In this context, the masked priming tions of the parafoveal preview reflect ‘what’ information was already technique deserves special consideration. In masked priming, a processed in the parafovea (e.g., orthography and/or phonology and/ target word is preceded by a briefly presented masked priming or morphology). stimulus (e.g., mln-melon). By manipulating the structural relation- There is ample evidence that high-level information, such as ships between prime and target (e.g., at the orthographic, phonolo- phonological [80,81], morphological [82,83], and lexical information gical, morphological, and other levels) for different exposure , influences very early aspects of the overall visual word durations (e.g., typically between 10 and 60 ms), researchers have recognition process. This evidence challenges the traditional claim observed different time courses of processing for different properties of temporal and structural modularity, according to which printed of printed words (e.g., orthographic and phonological representa- words are principally identified on the basis of orthographic tions). information alone in skilled readers (the underlying logic behind The rationale behind this experimental approach is that the minimal some researchers’ concept of the VWFA), with phonological and prime duration required to obtain a specific priming effect reflects the semantic information retrieved subsequently [64,85]. models for the development of a mechanistically explicit middle and anterior temporal, and the pars triangularis in theory of visual word recognition. the inferior frontal cortex. This notwithstanding, there is still a heated debate regarding the characterization of fMRI evidence suitable for ‘where’ but not for ‘when’ directionality of flow of information in these pathways (i.e., Many fMRI studies have investigated the brain circuits ‘when’ and ‘how’). Specifically, the literature is unsettled that underlie reading. Two points on which this research regarding the extent to which higher-level lexical repre- converge is that the left hemisphere plays a major role in sentations that are not necessarily orthographic modulate reading and the reading circuit consists of a network with the relatively early processing of orthographic information two major pathways: (i) a dorsal pathway including the (Box 3). occipital, supramarginal, and angular gyri, and the pre- One of the most relevant examples of such debates is the motor and pars opercularis in the inferior frontal cortex; role of the left fusiform gyrus, the putative visual word and (ii) a ventral pathway that integrates the left fusiform, form area (VWFA) [5,6]. From an anatomical processing (A) (B) Semancs Lexicon/semancs Orthographic lexicon Phonology Leers Phonemes Leers Visual features Visual features Visual input Visual input TRENDS in Cognitive Sciences Figure 1. Core architectural and functional assumptions of temporally modular feedforward versus interactive models of visual word recognition. (A) According to temporally modular feedforward models, visual orthographic information is processed in a set of distinct, hierarchically organized processing stages, such that each stage (e.g., activation of letter and orthographic lexical representations) occurs in a strictly feedforward – and in the strongest form, sequential – fashion. Critically, additional non- visual orthographic representations (e.g., phonology, semantics) are not accessed until orthographic access is complete, and/or if accessed before that point, higher-level representations never feed back to influence the orthographic computation. (B) According to interactive activation models , visual information continuously cascades throughout the entire orthographic–phonological–lexical–semantic network. This enables partially resolved phonological and lexical–semantic representations (among others) to feed back and provide constraints on other (lower) levels of representation in the network such as orthography. Note that additional intermediate levels of representation (e.g., letter clusters) have been suppressed for simplicity in both schematics, and that these are just two examples of each type of network (e.g., other feedforward theories suggest a direct sublexical input to phonology but are nevertheless feedforward). Unbroken blue lines denote feedforward connections; broken green lines denote feedback connections. 91 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 Box 2. Structural and temporal modularity, interactivity, and orthographic processing The main theoretical stand underlying the feedforward approach is top-down interactivity as long as it is constrained to occur after initial that pure orthographic models have substantial descriptive adequacy orthographic coding is complete. to account for a large set of (mostly behavioral) data in visual word The contrasting approach argues for full interactivity between recognition. One basic tenet in this feedforward view is that in lower- and higher-order representations at all processing levels. Here, principle, ‘feedback cannot improve performance at either the lexical the demarcation line beyond ‘when’ and ‘where’ ‘perceptual ortho- or prelexical level’ [86,p. 306] and a number of well-known graphic’ processing ends and ‘linguistic’ processing begins is blurred phenomena (e.g., word superiority effect) that have been traditionally. According to this view, high-level linguistic considerations that attributed to top-down feedback can indeed be explained parsimo- are not purely orthographic (e.g., how some letters correlate with niously in a feedforward manner. According to this view, the game of phonology and meaning, and how letter clusters are constrained by processing printed words is largely played in the court of ortho- lexical, morphological, and phonological structure) shape the dis- graphic processing, such that a significant part of the recognition tributional properties of letters in a given language, and the word process is determined by considering the surface structural properties recognition system learns these features to enable efficient (i.e., fast of the printed stimulus alone (i.e., letters, letter sequences). Interest- and accurate) reading in that language. Language-specific retinal– ingly, this position is consistent with the idea that the identification of perceptual learning effects (i.e., cross-linguistic difference in proces- visual forms in general and letter strings in particular can be achieved sing letters at different retinal eccentricities) suggest that reading through a low-level visual pattern recognition system shared by habits stemming from the overall structure of a language indeed humans and baboons [87–89]. affect the functional structure of early stages in the visual pathway, The strongest version of the feedforward view postulates structural and are thus compatible with this view. For example, frequently modularity, according to which orthographic processing is in encountered visual configurations result in perceptual learning that principle non-penetrable by other linguistic dimensions. Moreover, allows for rapid and efficient recognition of a word form, and these processing within the orthographic system proceeds bottom-up from configurations are influenced by the correlation of orthography with low-level features to full orthographic words. The weaker version phonology and meaning that are characteristic to a language. assumes temporal modularity and posits that the word recogni- Because different languages are characterized by different relations tion system is simply set so that the processing of printed words between orthography, phonology, and semantics (among other proceeds until an orthographic word unit is recognized; only when representations), it is argued that interactive models that allow for this is accomplished does the orthographic representation make phonological, morphological, and semantic information to come into contact with various other linguistic properties (e.g., phonology, play early on are better accounts for the substantial cross-linguistic morphology, semantics ). Note that this approach may allow for differences observed in early orthographic processing. perspective (i.e., a ‘where’ distinction), this brain region is two approaches provide very different views of reading: considered to be a relatively ‘early’ processing area. The The former is compatible with the notion of feedforward left fusiform gyrus is activated more for words or pseudo- temporal (and structural) modularity (Box 2), whereby words than for false fonts or consonant strings [7–10]. It is reading is considered to rely on a sequence of consecutive thus commonly accepted that the left fusiform is involved brain areas sensitive to a hierarchy of orthographic repre- in orthographic processing. There is a controversy, how- sentations (e.g., letters, letter clusters of increasing size) ever, regarding what specific information is represented in that culminates in recognition of a word. The latter con- this brain region and how sensitive it is to top-down siders reading as a fully interactive processing system information. One theoretical position is that the VWFA whereby higher-level linguistic information that is not is a prelexical hub, specific for written words, that com- necessarily orthographic modulates early perceptual putes and stores strictly visual and abstract prelexical orthographic processing. orthographic representations in a primarily feedforward Whereas proponents of the feedforward approach have manner [8,11,12]. Another theoretical position, however, relied on the argument that VWFA activation reflects a postulates that activation of the visual form area is modu- stage of orthographic processing that is immune to pho- lated by higher-order linguistic properties of stimuli such nological and semantic influences that come into play only as phonology, morphology, and semantics [13,14]. These later on [12,15,16], there is mounting evidence suggesting Box 3. Future explorations of interactivity using fMRI and MEG Several studies have shown intrinsic functional connections between pathways from vision to higher-order temporal lobe language areas. Broca’s area and ventral occipitotemporal regions [58,93–96] Anatomi- Thus, the full scope of interactivity (or lack thereof) between regions cal connections between frontal and occipital regions through the spanning the different pathways of the reading circuit should be more superior longitudinal fasciculus and/or the inferior fronto-occipital fully established. Finally, it is important to note that the dorsal and fasciculus have also been documented [97–105]. Taken together, these ventral pathways are not modular systems that operate independently findings provide a neurobiological platform for possible top-down of each other, but exchange information during visual word recognition effects from frontal areas, and thus offer interesting avenues for future [108,109]. In fact, structural connectivity between regions belonging to investigations. Moreover, it is important to keep in mind that readers do each of the two pathways (e.g., the posterior inferior temporal regions, not only activate the left fusiform and other regions of the ventral including the left fusiform, and the posterior superior temporal and pathway when reading. They also activate the dorsal pathway, inferior parietal regions, including the supramarginal gyrus) has been including regions such as the left supramarginal gyrus, the left superior documented. Moreover, functional connectivity between these temporal cortex, and the left inferior parietal cortex, and in particular regions has been shown in skilled readers but not in dyslexic the angular gyrus, which has a modulatory effect on the visual cortex individuals. Further studies are required to determine how brain [4,106]. Furthermore, there are other functional pathways in the reading regions falling along the ventral and dorsal pathways interact and circuit starting in the occipital cortex that do not necessarily involve the cooperate during visual word recognition, and how these interactions left fusiform. The reading circuit includes not just one but multiple relate to other similar processes such as object recognition. 92 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 that early print processing in the VWFA is modulated by (but see Rauschecker et al. for bilateral effects). In fact, high-level lexical information. For example, sensitivity to all this evidence supports the claim that at approximately high-level variables such as lexical frequency has been 150 ms from stimulus onset, the visual system responds observed in the left fusiform. Furthermore, the only to the frequency of letter combinations, and that VWFA was similarly activated when target words were lexical and phonological effects come into play much later preceded by masked printed-word primes or by masked [15,16,40]. As expounded below, however, higher-level lin- pictures (Box 1). Note that bidirectional information guistic information already exerts its influence at 100 ms flow whereby higher-order levels of processing constrain (i.e., before 170 ms) from stimulus onset. For instance, it feedforward assembly has also been proposed for object has been reported that early event-related potential (ERP) recognition [19–24]. Adopting the recycling hypothesis components in the range 100–200 ms are sensitive to (i.e., the neurocircuitry for visual object recognition lexical frequency [41–45]. Thus, from a simple time-scale is recycled to compute the representations necessary for perspective, an early marker of visual word recognition as human reading), simple parsimony considerations would revealed by ERP measures (but not fMRI measures) seems lead to the assumption of similar principles regarding the to be susceptible to modulation from higher-order lexical flow of information for visual object and visual word information. recognition. Another early marker of reading is the N250, which was Despite the above evidence, the debate regarding originally found to be sensitive to orthographic similarity whether processing of visual word recognition is feedfor- in combined masked priming and EEG studies [46,47]. ward or not is still as active as ever. This is because the However, subsequent studies have shown that N250 is critical distinction between the two conceptual approaches also modulated by lexical factors [48,49]. In particular, it regarding ‘how’ information flows in the circuits is mostly was found that this ERP component is sensitive not just to temporal in nature (‘early’ vs ‘late’). However, fMRI inte- letter identity but also to the phonological status of the grates processes over a relatively long period of time. Thus, letters, that is, whether letters are consonant or vowels although the timing of stimulus presentation can be well [46,50–52]. For example, Carreiras and colleagues showed controlled (e.g., masked priming, fMRI adaptation), the that masked subset priming of consonants (e.g., mln– temporal resolution of the blood-oxygen-level dependent melon) and masked full identity priming (e.g., melon– (BOLD) response is too slow to unequivocally distinguish melon) do not significantly differ from each other in the between activations that are feedforward versus feedback N250 component, whereas masked vowel subset priming (i.e., the ‘when’ question). This leads us to the inherent (e.g., aio–amigo) and masked identity priming (e.g., amigo– advantage of MEG and EEG. amigo) do. Because consonants are more lexically constraining than vowels in predicting word identity MEG and EEG: the dynamics of the time course , this effect demonstrates that top-down lexical infor- EEG and MEG are time-sensitive methods with a temporal mation modulates the N250 component. Note that the resolution in the range of milliseconds. Combined with same pattern of response is revealed in the later N400 appropriate designs, they can reveal the temporal order component and in RTs in behavioral experiments. This of the neural processes involved in visual word recognition, suggests that accumulated lexical information (and/or lex- tracing the time course from low-level visual perception to ical competition) that generates the masked prime has letter perception and word meaning. It is this time course exerted its full impact by 250 ms from stimulus onset. In that provides important evidence to adjudicate between fact, the dissociation found between transposed-letter different theoretical stances regarding the flow of informa- priming effects for word–word pairs (e.g., casual–causal) tion (feedforward vs feedback). Both techniques tap syn- and for nonword–word pairs (e.g., barin–brain) in the N250 chronized neuronal activity over time triggered by some component reinforces the hypothesis of high-order cognitive event in the brain. Unlike RTs, which give us lexical–semantic information constraining orthographic the end result of processing in the system as a whole (and form-level processing in the N250. not specifically of lexical processing; Box 1), both MEG and Consistent with sustained and early interactive coacti- EEG provide a continuous measure of the intermediate vation of a network of sites contributing to reading, Thesen events that have led to the final response. MEG also and colleagues found strong phase-locking from 170 to provides some spatiotemporal constraints, allowing for 400 ms between the left fusiform and more anterior lan- some synthesis of ‘when’ and ‘where’ information. guage areas when comparing words versus false fonts One of the earlier markers of visual word recognition is a using MEG and intracranial recording. Other recent left lateralized N150/N170 response that differentiates reports of very early neurobiological responses to phono- orthographic stimuli such as words and pseudowords from logical information in anterior areas are also consistent other stimuli such as symbols [27–29]. Selective responses with a top-down flow of information during visual word to letters in this time window have also been found in the recognition. Using MEG in a masked priming paradigm, inferior occipitotemporal cortex using intracranial record- Wheat and colleagues observed stronger responses to pseu- ings [30,31] and MEG [32,33], particularly for normal dohomophones than to orthographic control primes within readers but not for dyslexic children [34–37]. Thus, it 100 ms of target word onset in a cluster that included the has been proposed that the left-lateralized N170 could left inferior frontal gyrus (pars opercularis) and the pre- be an automatic response related to typical visual word central gyrus [55–57]. Note that a parallel pattern of recognition, and that it could be associated with the activa- activation found in the middle occipital gyrus suggests tion found using fMRI in the VWFA left fusiform gyrus that these regions could oscillate together during visual 93 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 word recognition at a very early stage. Thus, the inferior recognition will, by default, expect and generate some frontal gyrus may exert feedback control on regions degree of top-down influence for maximally accurate word involved in lower-level analysis of written words. In fact, identification. A more critical concern is if these top-down a recent study provides evidence of top-down feedback from influences are substantive enough to be theoretically sig- the inferior frontal gyrus to the left ventral occipitotem- nificant and may not be dismissed for reasons of parsimony poral cortex via dynamic causal modeling of MEG data [64–67]. Proficient reading involves optimization of effi-. Specifically, the researchers found that words (as ciency in addition to accuracy, that is, correct reading of compared to false fonts) activated the left inferior frontal words as quickly as possible. Thus, if a strictly feedforward gyrus. More importantly, they showed that feedback con- system could, in principle, enable highly efficient word nections from the inferior frontal gyrus to the left ventral recognition, under what circumstances would the brain occipitotemporal cortex within the first 200 ms provided choose to pay the price of waiting for additional top-down the best fit for the data relative to a model with only constraints because of inadequacies in a strictly feedfor- feedforward connectivity between these regions. One pos- ward signal ? sible explanation for this feedback is that the inferior Connectionist models offer several avenues for explor- frontal region sends higher-level information (e.g., phonol- ing these possibilities. One particularly important recent ogy) to constrain the representations computed in the left advance is the ability to incorporate additional neurobio- fusiform. Alternatively (or complementarily), these two logical constraints into standard connectionist models brain regions may be interacting bidirectionally as part (e.g., by specifying different subpopulations of inhibitory of a constraint network with as-yet underspecified graded and excitatory neurons) to simulate electrophysiological specialization across the different contributing brain and behavioral responses (B.C. Armstrong, Ph.D. thesis, regions. Although the specific representations and Carnegie Mellon University, 2012) [69,70]. For example, dynamics of the frontal–occipitotemporal areas remain Laszlo and Plaut showed how a model that instantiates to be elucidated, these data clearly challenge the notion these principles can generate and explain electrophysiolo- of temporal and structural modularity in orthographic gical dynamics corresponding to the N400 ERP component processing.. In addition, they were able to advance the field by offering an account of an important discrepant finding Biologically plausible connectionist modeling: a between behavioral and electrophysiological approaches: platform for advancing theories of visual word why the N400 ERP component is not sensitive to the recognition lexicality of the stimulus (e.g., words and pseudowords From the empirical data outlined above, it is clear that vs acronyms and illegal strings), whereas behavioral substantive theoretical advances will require an inte- responses are. Specifically, they showed that the initial grated understanding of the contributions of a large set settling dynamics, during which the prominent deflection of distributed representations stored in different brain typically associated with the N400 ERP component was regions that are accessed (at least initially) at different displayed, were primarily driven by the orthographic word- points in time as activity cascades throughout the brain. likeness of the stimulus (e.g., in terms of its orthographic Connectionist modeling offers a mechanistic platform that neighborhood). However, nonlinear settling dynamics in is ideally suited for these investigations because it allows the network caused a change in these activation patterns researchers to probe the ‘where’ and ‘when’ of visual word later on in processing, such that valid lexical types (words recognition and directly relate them to the questions of and acronyms) were more active than nonwords (pseudo- ‘what’ (representations) and ‘how’ (explicit computational words and illegal strings), consistent with typical beha- processing mechanisms) working in concert to enable the vioral lexical decision data. perception of written words. Moreover, models allow Laszlo and Armstrong further extended this work to researchers to explore the emergent properties of these account for how simple context effects (e.g., word repeti- systems and develop targeted empirical research agendas tion) modulate the N400 component associated with lex- for the future. ical–semantic access. This was accomplished via The basic capacities of connectionist networks as out- incorporation of a neuron-specific fatigue mechanism so lined above were keenly demonstrated via the interactive that recently fired neurons would not be able to fire at their activation model [59–61]. In this model, information from maximum rate for a brief period of time. This resulted in a low-level visual feature detectors flowed bottom-up to a reduction in N400-component amplitudes for stimuli in the lexical representation of whole words, while simulta- semantic representation, regardless of the lexical status of neously being able to flow top-down from higher levels of the character string input to the network. Moreover, they representation. This model can thus explain and generate were recently able to generate specific predictions regard- predictions regarding top-down influences related to word ing the power-frequency spectra that should be evoked by reading, such as the word superiority effect (i.e., the pro- words and nonwords (Laszlo and Armstrong, unpub- cessing advantage for letters embedded in words relative to lished), data that are increasingly influential in establish- isolated letters). ing the causal links between which brain regions influence From this work and the general mechanics of constraint one another and the temporal order (i.e., ‘when’) in which satisfaction systems , as well as the presence of bidir- this occurs (Figure 2) [58,70]. This work led to targeted ectional connectivity between brain regions that process insights into ‘what’ aspects of a word representation are different aspects of word representation (e.g., letters, pho- modulated by related context. Furthermore, this neurally nology, semantics), connectionist theories of visual word inspired account therefore presents an alternative 94  Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 (A) (B) Words Acronyms Pseudowords Illegal strings Semanc acvaon Recordings N400 window Semancs from simulaon Time (unit updates) Key: First presentaon Second presentaon Middle parietal electrode (Cz) Words Acronyms Pseudowords Illegal strings (C) N400 window Amplitude (µV) Recordings 5 from analogous 0 900 EEG/ERP experiment Visual input Time (ms) Key: First presentaon Second presentaon (D) (E) Semancs SG AG IF OC AT FG Visual input TRENDS in Cognitive Sciences Figure 2. Integration of insights from more biologically plausible connectionist models and neuroimaging data. Recent connectionist models that use large pools of excitatory neurons and small pools of inhibitory neurons (here, inhibitory subpopulations are denoted by –; all other neurons are excitatory), as well as sparse/weak distal connections (thin arrows) and dense/strong local connections (thick arrows), and emergent hidden representations that are not explicitly specified by the modeler (denoted by < > around the name of the approximate representation that should emerge in that pool) have led to major advances towards explaining and predicting behavioral and neuroimaging data with a computationally explicit theory. For example, Laszlo and Plaut and Laszlo and Armstrong used the architecture illustrated in (A) to generate (B) simulated ERP components that explain empirical ERP data from (C) an analogous experiment involving lexical types including words (e.g., DOG), acronyms (e.g., DVD), pseudowords (e.g., DOD), and illegal strings (e.g., XFQ) both on their initial presentation (thick lines) and when the item was repeated (thin lines). Note that the units for the model data are arbitrary and are omitted; for the empirical data, the x-axis ranges from 0 to 900 ms and the negative direction is plotted upwards. All empirical and simulation data were drawn from Laszlo and Armstrong , used with permission. These event-related potential data were collected from an electrode placed on a middle parietal location. This work suggests that (D) in a more comprehensive model that also contains orthographic and semantic representations (here the , , and labels denote intermediate pools of neurons that map between orthography, phonology, and semantics, respectively), it is in principle possible to study the activation dynamics and representations that emerge in (E) brain regions associated with different representations, of which a subset of the most critical regions are shown in a lateral cross-section of the left hemisphere. The color of the circles denotes the theoretical representations in the model that these regions might subserve. Abbreviations: IF, inferior frontal cortex; SG, supramarginal gyrus; AG, angular gyrus; AT, anterior temporal cortex; FG, fusiform gyrus (includes visual word form area, VWFA); OC, occipital cortex. explanation for the long-accepted verbal account of N400 direct assessment of the impact of different theoretical repetition effects , according to which reduced N400 assumptions. Moreover, by virtue of the domain-general amplitudes results from an (underspecified) facilitation nature of the framework, it is possible to naturally extend mechanism rather than a fatigue mechanism. these principles to the study of other levels of representa- With such explicit models in hand, it is possible to add or tion. For instance, these principles can be readily applied subtract different feedback connections and evaluate to study ERP components associated with earlier processes which of these models best captures empirical electrophy- related to visual word recognition (e.g., N170, N250) and siological data such as ERP waveform amplitudes and determine ‘when’ and ‘how’ these representations shape power over time in as parsimonious a manner as possible. other purportedly earlier processes. These models thus present an opportunity for an ‘experi- Combined with a domain-general learning theory and mental’ approach to theoretical development by allowing recent advances in ‘deep’ neural networks , it has 95 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 Box 4. Outstanding questions and future directions ‘black box’ theorizing regarding the internal mechanics of the brain that mediate between stimulus and response.  How does anatomical and functional connectivity, and conse- quently the temporal flow of information, evolve from preliterate Rather, these representations can now be monitored to skilled reading? directly and used to motivate specific theoretical claims  What are the functional and anatomical differences underlying about the intermediate internal representations and pro- reading disabilities, and how can their understanding can help the cesses that subserve visual word recognition. Like never implementation of remediation programs? before, it is therefore possible to achieve integrated the-  How are connectivity and interactivity modulated by different languages having different scripts, different orthography-to- ories of ‘what’, ‘when’, ‘where’, and ‘how’ visual words are phonology relations, and different morphological systems? represented and processed in the brain.  What do the constraints of possessing different orthographic, phonological, and semantic representations have on visual word Acknowledgments recognition in the case of bilingual and multilingual readers? We acknowledge support of M.C. by the European Research Council  How can more biologically plausible computational models (ERC-2011-ADG-295362) and by the Spanish Ministry of Economy and interact with empirical investigations to produce theories that Competitiveness CONSOLIDER-INGENIO2010 CSD2008-00048 and are mechanistically explicit, comprehensive, and parsimonious? PSI2012-31448; of B.C.A. by the Marie Curie program (PCIG13-2013-  To what extent will a neurobiological theory of visual word 618429); of R.F. by the NICHD (RO1 HD067364); of R.F. and M.C. recognition that considers the full patterns of brain connectivity (PO1HD 01994); and of M.P. by the Spanish Ministry of Economy and and interactivity provide insights into domain-general mechan- Competitiveness (PSI2011-26924). We thank S. Laszlo for discussion of isms shared by other related abilities such as object recognition? the modeling section. References recently become possible to train networks with multiple 1 Rogers, T.T. et al. (2004) Structure and deterioration of semantic intermediate ‘hidden’ layers whose input values or target memory: a neuropsychological and computational investigation. outputs are not explicitly set by the experimenter. Conse- Psychol. Rev. 111, 205–235 quently, researchers can step back from explicitly specify- 2 Plaut, D.C. (2002) Graded modality-specific specialisation in ing the exact nature of the representations that are semantics: a computational account of optic aphasia. Cogn. Neuropsychol. 19, 603–639 expected to mediate between a low-level visual form and 3 Plaut, D.C. and Behrmann, M. (2011) Complementary neural higher-order lexical information. Instead, they can simply representations for faces and words: a computational exploration. specify a more neutral input (e.g., a low-level visual input Cogn. Neuropsychol. 28, 251–275 associated with a word in a retinotopically centered coding 4 Price, C.J. (2012) A review and synthesis of the first 20 years of PET scheme) and observe how unsupervised learning and fMRI studies of heard speech, spoken language and reading. Neuroimage 62, 816–847 mechanics can gradually extract higher-order information 5 Cohen, L. et al. (2000) The visual word form area: spatial and temporal such as position-insensitive visual representations and characterization of an initial stage of reading in normal subjects and intermediate representations of different grain sizes posterior split-brain patients. Brain 123, 291–307. This approach allows empirical investigation of 6 Cohen, L. et al. (2002) Language-specific tuning of visual cortex? Functional properties of the visual word form area. Brain 125, whether particular intermediate representations assumed 1054–1069 to mediate between visual inputs and higher-order lexical 7 Baker, C.I. et al. (2007) Visual word processing and experiential outputs (e.g., bigrams, syllables) are in fact representa- origins of functional selectivity in human extrastriate cortex. Proc. tions that will emerge during learning, and how these Natl. Acad. Sci. U.S.A. 104, 9087–9092 representations are constrained via feedforward and feed- 8 Dehaene, S. et al. (2002) The visual word form area: a prelexical representation of visual words in the fusiform gyrus. Neuroreport 13, back constraints. 321–325 Taken together, biologically plausible connectionist 9 Glezer, L.S. et al. (2009) Evidence for highly selective neuronal tuning models are showing strong initial promise in being able to whole words in the visual word form area. Neuron 62, 199–204 to provide detailed and explicit mechanistic accounts of 10 Binder, J.R. et al. (2006) Tuning of the human left fusiform gyrus to ‘what’ and ‘where’ different types of representations are sublexical orthographic structure. Neuroimage 33, 739–748 11 Dehaene, S. et al. (2005) The neural code for written words: a proposal. stored [72,73], ‘when’ they interact, and ‘how’ the proces- Trends Cogn. Sci. 9, 335–341 sing mechanics of neurobiologically constrained computa- 12 Dehaene, S. and Cohen, L. (2011) The unique role of the visual word tions operate the way they do. Moreover, these models can form area in reading. Trends Cogn. Sci. 15, 254–262 be used not only as ‘process’ models of the typical mature 13 Price, C.J. and Devlin, J.T. (2003) The myth of the visual word form system but can also trace the development of lexical skills area. Neuroimage 19, 473–481 14 Price, C.J. and Devlin, J.T. (2011) The interactive account of ventral and top-down versus bottom-up interactivity , and can occipitotemporal contributions to reading. Trends Cogn. Sci. 15, 246– seamlessly facilitate studies of different types of reading 253 impairments. 15 Simos, P.G. et al. (2002) Brain mechanisms for reading words and pseudowords: an integrated approach. Cereb. Cortex 12, 297–305 16 Solomyak, O. and Marantz, A. (2010) Evidence for early Concluding remarks morphological decomposition in visual word recognition. J. Cogn. Old debates regarding the modularity – or lack thereof – of Neurosci. 22, 2042–2057 visual word recognition are back again with the advent of 17 Kronbichler, M. et al. (2004) The visual word form area and the techniques such as fMRI, EEG, and MEG. However, frequency with which words are encountered: evidence from a rekindling of these debates via research using such tech- parametric fMRI study. Neuroimage 21, 946–953 18 Kherif, F. et al. (2011) Automatic top-down processing explains niques offers an opportunity for unique theoretical advance common left occipito-temporal responses to visual words and that was not possible via behavioral investigations alone, objects. Cereb. Cortex 21, 103–114 and also enables the investigation of important new ques- 19 Bar, M. et al. (2006) Top-down facilitation of visual recognition. Proc. tions (Box 4). No longer are researchers constrained to Natl. Acad. Sci. U.S.A. 103, 449–454 96 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 20 Bar, M. (2007) The proactive brain: using analogies and associations priming paradigm using event-related potentials. Brain Res. 1180, to generate predictions. Trends Cogn. Sci. 11, 280–289 39–58 21 Kveraga, K. et al. (2007) Magnocellular projections as the trigger of 48 Duñabeitia, J.A. et al. (2009) N250 effects for letter transpositions top-down facilitation in recognition. J. Neurosci. 27, 13232–13240 depend on lexicality: ‘casual’ or ‘causal’? Neuroreport 20, 381–387 22 Gazzaley, A. et al. (2007) Functional interactions between prefrontal 49 Massol, S. et al. (2011) When less is more: feedback, priming, and the and visual association cortex contribute to top-down modulation of pseudoword superiority effect. Brain Res. 1386, 153–164 visual processing. Cereb. Cortex 17 (Suppl. 1), i125–i135 50 Carreiras, M. et al. (2009) Are vowels and consonants processed 23 Zanto, T.P. et al. (2011) Causal role of the prefrontal cortex in top- differently? Event-related potential evidence with a delayed letter down modulation of visual processing and working memory. Nat. paradigm. J. Cogn. Neurosci. 21, 275–288 Neurosci. 14, 656–661 51 Carreiras, M. et al. (2007) ERP correlates of transposed-letter 24 Bar, M. (2003) A cortical mechanism for triggering top-down similarity effects: are consonants processed differently from facilitation in visual object recognition. J. Cogn. Neurosci. 15, 600–609 vowels? Neurosci. Lett. 419, 219–224 25 Logothetis, N.K. (2008) What we can do and what we cannot do with 52 Carreiras, M. et al. (2009) ERP correlates of transposed-letter priming fMRI. Nature 453, 869–878 effects: the role of vowels versus consonants. Psychophysiology 46, 34– 26 Hämäläinen, M. et al. (1993) Magnetoencephalography – theory, 42 instrumentation, and applications to noninvasive studies of the 53 Dunabeitia, J.A. and Carreiras, M. (2011) The relative position working human brain. Rev. Mod. Phys. 65, 413 priming effect depends on whether letters are vowels or 27 Dunabeitia, J.A. et al. (2012) Differential sensitivity of letters, consonants. J. Exp. Psychol. Learn. Mem. Cogn. 37, 1143–1163 numbers, and symbols to character transpositions. J. Cogn. 54 Thesen, T. et al. (2012) Sequential then interactive processing of Neurosci. 24, 1610–1624 letters and words in the left fusiform gyrus. Nat. Commun. 3, 1284 28 Bentin, S. et al. (1999) ERP manifestations of processing printed 55 Cornelissen, P.L. et al. (2009) Activation of the left inferior frontal words at different psycholinguistic levels: time course and scalp gyrus in the first 200 ms of reading: evidence from distribution. J. Cogn. Neurosci. 11, 235–260 magnetoencephalography (MEG). PLoS ONE 4, e5359 29 Maurer, U. et al. (2005) Fast, visual specialization for reading in 56 Pammer, K. et al. (2004) Visual word recognition: the first half second. English revealed by the topography of the N170 ERP response. Behav. Neuroimage 22, 1819–1825 Brain Funct. 1, 13 57 Wheat, K.L. et al. (2010) During visual word recognition, phonology is 30 Allison, T. et al. (1994) Human extrastriate visual cortex and the accessed within 100 ms and may be mediated by a speech production perception of faces, words, numbers, and colors. Cereb. Cortex 4, 544– code: evidence from magnetoencephalography. J. Neurosci. 30, 5229– 554 5233 31 Nobre, A.C. et al. (1994) Word recognition in the human inferior 58 Woodhead, Z.V. et al. (2012) Reading front to back: MEG evidence for temporal lobe. Nature 372, 260–263 early feedback effects during word recognition. Cereb. Cortex http:// 32 Tarkiainen, A. et al. (1999) Dynamics of letter string perception in the dx.doi.org/10.1093/cercor/bhs365 human occipitotemporal cortex. Brain 122, 2119–2132 59 McClelland, J.L. and Rumelhart, D.E. (1981) An interactive 33 Tarkiainen, A. et al. (2002) Dynamics of visual feature analysis and activation model of context effects in letter perception: I. An object-level processing in face versus letter-string perception. Brain account of basic findings. Psychol. Rev. 88, 375 125, 1125–1136 60 Seidenberg, M.S. and McClelland, J.L. (1989) A distributed, 34 Helenius, P. et al. (1999) Dissociation of normal feature analysis and developmental model of word recognition and naming. Psychol. deficient processing of letter-strings in dyslexic adults. Cereb. Cortex Rev. 96, 523–568 9, 476–483 61 Plaut, D.C. et al. (1996) Understanding normal and impaired word 35 Simos, P.G. et al. (2002) Dyslexia-specific brain activation profile reading: computational principles in quasi-regular domains. Psychol. becomes normal following successful remedial training. Neurology Rev. 103, 56–115 58, 1203–1213 62 Reicher, G.M. (1969) Perceptual recognition as a function of 36 Simos, P.G. et al. (2005) Early development of neurophysiological meaningfulness of stimulus material. J. Exp. Psychol. 81, 275–280 processes involved in normal reading and reading disability: a 63 Ackley, D.H. et al. (1985) A learning algorithm for Boltzmann magnetic source imaging study. Neuropsychology 19, 787–798 machines. Cogn. Sci. 9, 147–169 37 Simos, P.G. et al. (2007) Altering the brain circuits for reading through 64 Davis, C.J. (2012) Developing a universal model of reading intervention: a magnetic source imaging study. Neuropsychology 21, necessitates cracking the orthographic code. Behav. Brain Sci. 1, 485–496 21–22 38 Brem, S. et al. (2010) Brain sensitivity to print emerges when children 65 Cohen, L. and Dehaene, S. (2004) Specialization within the ventral learn letter-speech sound correspondences. Proc. Natl. Acad. Sci. stream: the case for the visual word form area. Neuroimage 22, 466– U.S.A. 107, 7939–7944 476 39 Rauschecker, A.M. et al. (2012) Position sensitivity in the visual word 66 Borowsky, R. and Besner, D. (1993) Visual word recognition: a form area. Proc. Natl. Acad. Sci. U.S.A. 109, E1568–E1577 multistage activation model. J. Exp. Psychol. Learn. Mem. Cogn. 40 Szwed, M. et al. (2012) Towards a universal neurobiological 19, 813–840 architecture for learning to read. Behav. Brain Sci. 35, 308 67 Borowsky, R. and Besner, D. (2006) Parallel distributed processing 41 Assadollahi, R. and Pulvermuller, F. (2003) Early influences of word and lexical-semantic effects in visual word recognition: are a few length and frequency: a group study using MEG. Neuroreport 14, stages necessary? Psychol. Rev. 113, 181–195 1183–1187 68 Plaut, D.C. and Shallice, T. (1993) Deep dyslexia: a case study of 42 Carreiras, M. et al. (2005) Early event-related potential effects of connectionist neuropsychology. Cogn. Neuropsychol. 10, 377–500 syllabic processing during visual word recognition. J. Cogn. Neurosci. 69 Laszlo, S. and Plaut, D.C. (2012) A neurally plausible parallel 17, 1803–1817 distributed processing model of event-related potential word 43 Dambacher, M. et al. (2006) Frequency and predictability effects on reading data. Brain Lang. 120, 271–281 event-related potentials during reading. Brain Res. 1084, 89–103 70 Laszlo, S. and Armstrong, B.C. (2013) Applying the dynamics of post- 44 Hauk, O. et al. (2006) The time course of visual word recognition as synaptic potentials to individual units in simulation of temporally revealed by linear regression analysis of ERP data. Neuroimage 30, extended ERP reading data. In Proceedings of the 35th Annual 1383–1400 Conference of the Cognitive Science Society (Knauff, M. et al., eds), 45 Sereno, S. (2003) Measuring word recognition in reading: eye pp. 2826–2831, Cognitive Science Society movements and event-related potentials. Trends Cogn. Sci. 7, 489–493 71 Rugg, M.D. (1985) The effects of semantic priming and work repetition 46 Carreiras, M. et al. (2009) Consonants and vowels contribute on event-related potentials. Psychophysiology 22, 642–647 differently to visual word recognition: ERPs of relative position 72 Hinton, G.E. and Salakhutdinov, R.R. (2006) Reducing the priming. Cereb. Cortex 19, 2659–2670 dimensionality of data with neural networks. Science 313, 504–507 47 Holcomb, P.J. and Grainger, J. (2007) Exploring the temporal 73 Di Bono, M.G. and Zorzi, M. (2013) Deep generative learning of dynamics of visual word recognition in the masked repetition location-invariant visual word recognition. Front. Psychol. 4, 635 97 Review Trends in Cognitive Sciences February 2014, Vol. 18, No. 2 74 Harm, M.W. and Seidenberg, M.S. (1999) Phonology, reading 94 Bitan, T. et al. (2005) Shifts of effective connectivity within a acquisition, and dyslexia: insights from connectionist models. language network during rhyming and spelling. J. Neurosci. 25, Psychol. Rev. 106, 491–528 5397–5403 75 Forster, K.I. and Davis, C. (1984) Repetition priming and frequency 95 Koyama, M.S. et al. (2010) Reading networks at rest. Cereb. Cortex 20, attenuation in lexical access. J. Exp. Psychol. Learn. Mem. Cogn. 10, 680 2549–2559 76 Ferrand, L. and Grainger, J. (1994) Effects of orthography are 96 Mechelli, A. et al. (2005) Dissociating reading processes on the basis of independent of phonology in masked form priming. Q. J. Exp. neuronal interactions. J. Cogn. Neurosci. 17, 1753–1765 Psychol. 47, 365–382 97 Wakana, S. et al. (2004) Fiber tract-based atlas of human white 77 Tzur, B. and Frost, R. (2007) SOA does not reveal the absolute time matter anatomy. Radiology 230, 77–87 course of cognitive processing in fast priming experiments. J. Mem. 98 Barbas, H. (2000) Connections underlying the synthesis of cognition, Lang. 56, 321–335 memory, and emotion in primate prefrontal cortices. Brain Res. Bull. 78 Rayner, K. (1998) Eye movements in reading and information 52, 319–330 processing: 20 years of research. Psychol. Bull. 124, 372 99 Bernal, B. and Altman, N. (2010) The connectivity of the superior 79 Bélanger, N.N. et al. (2013) Orthographic and phonological preview longitudinal fasciculus: a tractography DTI study. Magn. Reson. benefits: parafoveal processing in skilled and less-skilled deaf Imaging 28, 217–225 readers. Q. J. Exp. Psychol. 66, 2237–2252 100 Kringelbach, M.L. (2004) and E. T. Rolls, The functional 80 Frost, R. (1998) Toward a strong phonological theory of visual word neuroanatomy of the human orbitofrontal cortex: evidence from recognition: true issues and false trails. Psychol. Bull. 123, 71 neuroimaging and neuropsychology. Prog. Neurobiol. 72, 341–372 81 Rastle, K. and Brysbaert, M. (2006) Masked phonological priming 101 Makris, N. et al. (2005) Segmentation of subcomponents within the effects in English: are they real? Do they matter? Cogn. Psychol. 53, superior longitudinal fascicle in humans: a quantitative, in vivo, DT- 97–145 MRI study. Cereb. Cortex 15, 854–869 82 Deutsch, A. et al. (2003) Early morphological effects in reading: 102 Bürgel, U. et al. (2006) White matter fiber tracts of the human brain: evidence from parafoveal preview benefit in Hebrew. Psychon. Bull. three-dimensional mapping at microscopic resolution, topography Rev. 10, 415–422 and intersubject variability. Neuroimage 29, 1092–1105 83 Deutsch, A. et al. (2005) Morphological parafoveal preview benefit effects 103 Yeterian, E. and Pandya, D. (2010) Fiber pathways and cortical in reading: evidence from Hebrew. Lang. Cogn. Process. 20, 341–371 connections of preoccipital areas in rhesus monkeys. J. Comp. 84 Dunabeitia, J.A. et al. (2009) There is no clam with coats in the calm Neurol. 518, 3725–3751 coast: delimiting the transposed-letter priming effect. Q. J. Exp. 104 Yeterian, E.H. et al. (2012) The cortical connectivity of the prefrontal Psychol. 62, 1930–1947 cortex in the monkey brain. Cortex 48, 58–81 85 Frost, R. (2012) A universal approach to modeling visual word 105 Yeatman, J.D. et al. (2013) Anatomy of the visual word form area: recognition and reading: not only possible, but also inevitable. adjacent cortical circuits and long-range white matter connections. Behav. Brain Sci. 35, 310–329 Brain Lang. 125, 146–155 86 Norris, D. et al. (2000) Merging information in speech recognition: 106 Carreiras, M. et al. (2009) An anatomical signature for literacy. feedback is never necessary. Behav. Brain Sci. 23, 299–325 Nature 461, 983–986 87 Grainger, J. et al. (2012) Orthographic processing in baboons (Papio 107 Richardson, F.M. et al. (2011) Multiple routes from occipital to papio). Science 336, 245–248 temporal cortices during reading. J. Neurosci. 31, 8239–8247 88 Ziegler, J.C. et al. (2013) What can we learn from humans about 108 Rosazza, C. et al. (2009) Early involvement of dorsal and ventral orthographic processing in monkeys? A reply to Frost and Keuleers pathways in visual word recognition: an ERP study. Brain Res. 1272, (2013). Psychol. Sci. 24, 1870–1871 32–44 89 Frost, R. and Keuleers, E. (2013) What can we learn from monkeys 109 Zachariou, V. et al. (2014) Ventral and dorsal visual stream about orthographic processing in humans? A Reply to Ziegler et al.. contributions to the perception of object shape and object location. Psychol. Sci. 22, 1868–1869 J. Cogn. Neurosci. 26, 189–209 90 Andrews, S. (2006) All about words: a lexicalist perspective on 110 Thiebaut de Schotten, M. et al. (2012) Learning to read improves the reading. In From Ink Marks to Ideas: Current Issues in Lexical structure of the arcuate fasciculus. Cereb. Cortex http://dx.doi.org/ Processing (Andrews, S., ed.), pp. 314–348, Psychology Press 10.1093/cercor/bhs383 91 Nazir, T.A. et al. (2004) Reading habits, perceptual learning, and 111 Vogel, A.C. et al. (2012) The putative visual word form area is recognition of printed words. Brain Lang. 88, 294–311 functionally connected to the dorsal attention network. Cereb. 92 Frost, R. et al. (2012) Towards a universal model of reading. Behav. Cortex 22, 537–549 Brain Sci. 35, 263 112 van der Mark, S. et al. (2011) The left occipitotemporal system in 93 Bitan, T. et al. (2007) Developmental changes in activation and reading: disruption of focal fMRI connectivity to left inferior frontal effective connectivity in phonological processing. Neuroimage 38, and inferior parietal language areas in children with dyslexia. 564–575 Neuroimage 54, 2426–2436 98