Chapter 21: The Constructive Nature of Visual Processing PDF

21 The Constructive Nature of Visual Processing Visual Perception Is a Constructive Process complex visual environments is an extraordinary com- Visual Processing Is Mediated by the Geniculostriate putational achievement that artifici...

21 The Constructive Nature of Visual Processing Visual Perception Is a Constructive Process complex visual environments is an extraordinary com- Visual Processing Is Mediated by the Geniculostriate putational achievement that artificial vision systems Pathway have yet to duplicate. Vision is used not only for object Form, Color, Motion, and Depth Are Processed in Discrete recognition but also for guiding our movements, and Areas of the Cerebral Cortex these separate functions are mediated by at least two The Receptive Fields of Neurons at Successive Relays in the parallel and interacting pathways. Visual Pathway Provide Clues to How the Brain Analyzes The existence of parallel pathways in the visual Visual Form system raises one of the central questions of cognition, The Visual Cortex Is Organized Into Columns of the binding problem: How are different types of infor- Specialized Neurons mation carried by discrete pathways brought together Intrinsic Cortical Circuits Transform Neural Information into a coherent visual image? Visual Information Is Represented by a Variety of Neural Codes Visual Perception Is a Constructive Process Highlights Vision is often incorrectly compared to the operation of a camera. A camera simply reproduces point-by-point the light intensities in one plane of the visual field. We are so familiar with seeing, that it takes a leap of imagi- The visual system, in contrast, does something funda- nation to realize that there are problems to be solved. But mentally different. It interprets the scene and parses it consider it. We are given tiny distorted upside-down images into distinct components, separating foreground from in the eyes and we see separate solid objects in surround- background. The visual system is less accurate than a ing space. From the patterns of stimulation on the retina we camera at certain tasks, such as quantifying the abso- perceive the world of objects and this is nothing short of a miracle. lute level of brightness or identifying spectral color. However, it excels at tasks such as recognizing a charg- —Richard L. Gregory, Eye and Brain, 1966 ing animal (or a speeding car) whether in bright sun- light or at dusk, in an open field or partly occluded by trees (or other cars). And it does so rapidly to let the viewer respond and, if necessary, escape. M ost of our impressions of the world and A potentially unifying insight reconciling the vis- our memories of it are based on sight. Yet the ual system’s remarkable ability to grasp the bigger pic- mechanisms that underlie vision are not at all ture with its inaccuracy regarding details of the input obvious. How do we perceive form and movement? is that vision is a biological process that has evolved How do we distinguish colors? Identifying objects in in step with our ecological needs. This insight helps Kandel-Ch21_0496-0520.indd 496 20/01/21 2:43 PM  Chapter 21 / The Constructive Nature of Visual Processing  497 explain why the visual system is so efficient at extract- We see a uniform six-by-six array of dots as either ing useful information such as the identities of objects rows or columns because of the visual system’s ten- independent of lighting conditions, while giving less dency to impose a pattern. If the dots in each row are importance to aspects like the exact nature of the ambi- similar, we are more likely to see a pattern of alternat- ent light. Moreover, vision does so using previously ing rows (Figure 21–1A). If the dots in each column learned rules about the structure of the world. Some are closer together than those in the rows, we are more of these rules appeared to have become wired into our disposed to see a pattern of columns (Figure 21–1B). neural circuits over the course of evolution. Others are The principle of good continuation is an important more plastic and help the brain guess at the scene pre- basis for linking line elements into unified shapes sented to the eyes based on the individual’s past expe- (Figure 21–1C). It is also seen in the phenomenon of rience. This complex, purposeful processing happens contour saliency, whereby smooth contours tend to at all levels of the visual system. It starts even at the pop out from complex backgrounds (Figure 21–1D). retina, which is specialized to pick out object bounda- The Gestalt features that we are disposed to pick out ries rather than creating a point-by-point representa- are also ones that characterize objects in natural scenes. tion of uniform surfaces. Statistical studies of natural scenes show that object This constructive nature of visual perception has boundaries are likely to contain visual elements that only recently been fully appreciated. Earlier thinking lie in close proximity, are continuous across intersec- about sensory perception was greatly influenced by tions, or form smooth contours. It is tempting to specu- the British empiricist philosophers, notably John Locke, late that the formal features of objects in natural scenes David Hume, and George Berkeley, who thought of per- created evolutionary pressure on our visual systems to ception as an atomistic process in which simple sensory develop neural circuits that have made us sensitive to elements, such as color, shape, and brightness, were those features. assembled in an additive way, component by compo- Separating the figure and background in a visual nent. The modern view that perception is an active and scene is an important step in object recognition. At dif- creative process that involves more than just the infor- ferent moments, the same elements in the visual field mation provided to the retina has its roots in the phi- can be organized into a recognizable figure or serve as losophy of Immanuel Kant and was developed in detail part of the background for other figures (Figure 21–2). in the early 20th century by the German psychologists This process of segmentation relies not only on certain Max Wertheimer, Kurt Koffka, and Wolfgang Köhler, geometric principles, but also on cognitive influences who founded the school of Gestalt psychology. such as attention and expectation. Thus, a priming The German term Gestalt means configuration or stimulus or an internal representation of object shape form. The central idea of the Gestalt psychologists is can facilitate the association of visual elements into a that what we see about a stimulus—the perceptual unified percept (Figure 21–3). This internal representa- interpretation we make of any visual object—depends tion can take many different forms reflecting the wide not just on the properties of the stimulus but also on range of time scales and mechanisms of neural encod- its context, on other features in the visual field. The ing. It could consist of transient reverberating spiking Gestalt psychologists argued that the visual system activity selective to a shape or a decision, lasting a frac- processes sensory information about the shape, color, tion of a second, or the selective modulation of synap- distance, and movement of objects according to com- tic weights during a particular context of a task or an putational rules inherent in the system. The brain has a expected shape, or circuit changes that could comprise way of looking at the world, a set of expectations that a long-term memory. derives in part from experience and in part from built- The brain analyzes a visual scene at three levels: low, in neural wiring. intermediate, and high (Figure 21–4). At the lowest level, Max Wertheimer wrote: “There are entities where which we consider in the next chapter (Chapter 22), the behavior of the whole cannot be derived from its visual attributes such as local contrast, orientation, individual elements nor from the way these elements color, and movement are discriminated. The interme- fit together; rather the opposite is true: the proper- diate level involves analysis of the layout of scenes ties of any of the parts are determined by the intrinsic and of surface properties, parsing the visual image structural laws of the whole.” In the early part of the into surfaces and global contours, and distinguishing 20th century, the Gestalt psychologists worked out the foreground from background (Chapter 23). The high- laws of perception that determine how we group ele- est level involves object recognition (Chapter 24). Once ments in the visual scene, including similarity, proxim- a scene has been parsed by the brain and objects rec- ity, and good continuation. ognized, the objects can be matched with memories of Kandel-Ch21_0496-0520.indd 497 20/01/21 2:43 PM A Similarity C Good continuation D Contour saliency c d b a B Proximity b a c Figure 21–1 Organizational rules of visual perception. To C. Line segments are perceptually linked when they are col- link the elements of a visual scene into unified percepts, the linear. In the top set of lines, one is more likely to see line seg- visual system relies on organizational rules such as similarity, ment a as belonging with c rather than d. In the bottom set, a proximity, and good continuation. and c are perceptually linked because they maintain the same A. Because the dots in alternating rows have the same color, curvature, whereas a and b appear to be discontinuous. an overall pattern of blue and white rows is perceived. D. The principle of good continuation is also seen in contour saliency. On the right, a smooth contour of line elements pops B. The dots in the columns are closer together than those in out from the background, whereas the jagged contour on the the rows, leading to the perception of columns. left is lost in the background. (Adapted, with permission, from Field, Hayes, and Hess 1993. Copyright © 1993 Elsevier Ltd.) Figure 21–2 Object recognition depends on segmentation of a scene into foreground and background. Recognition of the white salamanders in this image depends on the brain “locating” the white salamanders in the foreground and the brown and black salamanders in the background. The image also illustrates the role of higher influences in segmentation: One can consciously select any of the three colors as the foreground. (Reproduced, with permission, from M.C. Escher’s “Symmetry Drawing E56” © 2010 The M.C. Escher Company- Holland. All rights reserved. www.mcescher.com.)  Chapter 21 / The Constructive Nature of Visual Processing  499 Figure 21–3 Expectation and perceptual task play a critical role in what is seen. It is difficult to separate the dark and white patches in this figure into foreground and background without additional information. This figure immediately becomes recogniz- able after viewing the priming image on page 501. In this example, higher-order representations of shape guide lower-order processes of surface segmentation. (Reproduced, with permission, from Porter 1954. Copyright 1954 by the Board of Trustees of the University of Illinois. Used with permission of the University of Illinois Press.) shapes and their associated meanings. Vision also has green, and blue versus yellow. These features are com- an important role in guiding body movement, particu- puted by different populations of specialized neural larly hand movement (Chapter 25). circuits forming independent processing modules that In vision, as in other cognitive operations, vari- separately cover the visual field. Thus, each point in ous features—motion, depth, form, and color—occur the visual field is processed in multiple channels that together in a unified percept. This unity is achieved extract distinct aspects of the visual input simultane- not by one hierarchical neural system but by multiple ously and in parallel. These parallel streams are then areas in the brain that are fed by parallel but interact- sent out along the axons of the retinal ganglion cells, ing neural pathways. Because distributed processing the projection neurons of the retina, which form the is one of the main organizational principles in the neu- optic nerves. robiology of vision, one must have a grasp of the ana- From the eye, the optic nerve extends to a midline tomical pathways of the visual system to understand crossing point, the optic chiasm. Beyond the chiasm, fully the physiological description of visual processing the fibers from each temporal hemiretina proceed to in later chapters. the ipsilateral hemisphere along the ipsilateral optic In this chapter, we lay the foundation for under- tract; fibers from the nasal hemiretinas cross to the standing the neural circuitry and organizational princi- contralateral hemisphere along the contralateral optic ples of the visual pathways. These principles apply quite tract (Figure 21–5). Because the temporal hemiretina of broadly and are relevant not only for the multiple areas of one eye sees the same half of the visual field (hemi- the brain concerned with vision but also for other types of field) as the nasal hemiretina of the other, the partial sensory information processing by the brain. decussation of fibers at the chiasm ensures that all the information about each hemifield is processed in the visual cortex of the contralateral hemisphere. The Visual Processing Is Mediated by the layout of the pathway also forms the basis for useful Geniculostriate Pathway diagnostic information. As a consequence of the par- ticular anatomy of this visual pathway, lesions at differ- The brain’s analysis of visual scenes begins in the two ent points along the pathway lead to visual deficits with retinas, which transform visual input using a strategy different geometric shapes (Figure 21–5) that can be dis- of parallel processing (Chapter 22). This important tinguished reliably through clinical examination. The neural computation strategy is utilized at all stages of deficit could be entirely monocular; if present in both the visual pathway as well as in other sensory areas. eyes, it could affect noncorresponding or correspond- The pixel-like bits of visual input falling on individual ing parts of the visual field in the two eyes; it could be photoreceptors—rods and cones—are analyzed by restricted to either the upper or the lower visual field or retinal circuits to extract some 20 local features, such may extend into both, etc. Thus, the shape of the deficit as the local contrasts of dark versus light, red versus could give valuable clues about type and location of the Kandel-Ch21_0496-0520.indd 499 20/01/21 2:43 PM 500  Part IV / Perception Low-level Intermediate-level High-level processing processing processing Contour integration Orientation Surface properties Color Shape discrimination Contrast Object identification Surface depth Disparity Surface segmentation Movement direction Object motion/ Shape from kinematic cues Figure 21–4 A visual scene is analyzed at three levels. Simple is integrated into global contours (contour integration), and attributes of the visual environment are analyzed (low-level surface shape is identified from shading and kinematic cues. processing), and these low-level features are then used to Finally, surfaces and contours are used to identify the object parse the visual scene (intermediate-level processing): Local (high-level processing). (M.C. Escher’s “Day and Night”. visual features are assembled into surfaces, objects are segre- © 2020 The M.C. Escher Company—The Netherlands. All rights gated from background (surface segmentation), local orientation reserved. www.mcescher.com)  Chapter 21 / The Constructive Nature of Visual Processing  501 Priming image for Figure 21–3 underlying nerve damage or occlusion (ranging from the primary visual cortex. But the LGN is not simply a optic nerve degeneration, such as due to multiple scle- relay; the retinal information it receives can be strongly rosis, to tumors, strokes, or physical trauma). modulated by attention and arousal through inhibi- Beyond the optic chiasm, the axons from nasal and tory connections to this brain region and by feedback temporal hemiretinas carrying input from one hemi- from the visual cortex. field join in the optic tract, which extends to the lateral The primary visual pathway is also called the geniculate nucleus (LGN) of the thalamus. The LGN in geniculostriate pathway (Figure 21–6A) because it primates consists of six primary layers: four parvocel- passes through the LGN on its way to the primary visual lular (Latin Parvus, small) and two magnocellular, each cortex (V1), also known as the striate cortex because paired with a thin but dense intercalated or koniocel- of the myelin-rich stripe that runs through its middle lular (Greek konio, dust) layer (see Figure 21–14). The layers. A second pathway extends from the retina to term “koniocellular” refers to the substantially smaller the pretectal area of the midbrain, where neurons medi- cell bodies in these layers relative to those of magno- ate the pupillary reflexes that control the amount of light cellular or parvocellular layers. The parallel channels entering the eyes (Figure 21–6B). A third pathway from established in the retinas remain anatomically segre- the retina runs to the superior colliculus and is important gated through the LGN. Parvocellular layers get input in controlling eye movements. This pathway continues from the midget retinal ganglion cells, which are the to the pontine formation in the brain stem and then to most numerous in the primate retina (~70%) and carry the extraocular motor nuclei (Figure 21–6C). red-green opponent information (Chapter 22). Mag- Each LGN projects to the primary visual cortex nocellular layers get achromatic contrast information through a pathway known as the optic radiation. These from the parasol ganglion cells (~10%). Koniocellular afferent fibers form a complete neural map of the con- layers get input from the small and large bistratified tralateral visual field in the primary visual cortex. Beyond ganglion cells, carrying blue-yellow information, that the striate cortex lie the extrastriate areas, a set of higher- together make up the third most populous set of retinal order visual areas that are also organized as neural maps projections to the LGN (~8%). Koniocellular layers also of the visual field. The preservation of the spatial arrange- get inputs from a number of other numerically much ment of inputs from the retina is called retinotopy, and a smaller classes of retinal ganglion cells. neural map of the visual field is described as retinotopic Each geniculate layer receives input from either or having a retinotopic frame of reference. the ipsilateral or the contralateral eye (see Figure The primary visual cortex constitutes the first 21–12) but is aligned so as to come from a matching level of cortical processing of visual information. From region of the contralateral hemifield. Thus, they form there, information is transmitted over two major path- a set of concordant maps stacked atop one another. ways. A ventral pathway into the temporal lobe carries The thalamic neurons then relay retinal information to information about what the stimulus is, and a dorsal Kandel-Ch21_0496-0520.indd 501 20/01/21 2:43 PM  502  Part IV / Perception Left visual field Right visual field Binocular zone Defects in visual field of Left eye Right eye Monocular Monocular 1 crescent crescent 2 Left Right 1 3 2 Optic nerve Optic chiasm Optic 4 3 tract 4 5 Lateral geniculate body Optic radiation 6 6 5+6 5 Figure 21–5 Representation of the visual field along the vis- 2. A lesion of the optic chiasm causes a loss of vision in the ual pathway. Each eye sees most of the visual field, with the temporal half of each visual hemifield (bitemporal hemianopsia). exception of a portion of the peripheral visual field known as 3. A lesion of the optic tract causes a loss of vision in the oppo- the monocular crescent. The axons of retinal neurons (ganglion site half of the visual hemifield (contralateral hemianopsia). cells) carry information from each visual hemifield along the 4. A lesion of the optic radiation fibers that curve into the tem- optic nerve up to the optic chiasm, where fibers from the nasal poral lobe (Meyer’s loop) causes loss of vision in the upper hemiretina cross to the opposite hemisphere. Fibers from the quadrant of the contralateral visual hemifield in both eyes temporal hemiretina stay on the same side, joining the fibers (upper contralateral quadrantic anopsia). from the nasal hemiretina of the contralateral eye to form the optic tract. The optic tract carries information from the opposite 5, 6. Partial lesions of the visual cortex lead to deficits in por- visual hemifield originating in both eyes and projects into the tions of the contralateral visual hemifield. For example, a lesion lateral geniculate nucleus. Cells in this nucleus send their axons in the upper bank of the calcarine sulcus (5) causes a partial along the optic radiation to the primary visual cortex. deficit in the inferior quadrant, while a lesion in the lower bank Lesions along the visual pathway produce specific visual (6) causes a partial deficit in the superior quadrant. The central field deficits, as shown on the right: area of the visual field tends to be unaffected by cortical lesions because of the extent of the representation of the fovea and 1. A lesion of an optic nerve causes a total loss of vision in one the duplicate representation of the vertical meridian in the eye. hemispheres. pathway into the parietal lobe carries information of objects spanning the vertical meridian by linking the about where the stimulus is, information that is critical cortical areas that represent opposite hemifields. for guiding movement. A major fiber bundle called the corpus callosum connects the two hemispheres, transmitting informa- Form, Color, Motion, and Depth Are Processed tion across the midline. The primary visual cortex in in Discrete Areas of the Cerebral Cortex each hemisphere represents slightly more than half the visual field, with the two hemifield representations In the late 19th and early 20th centuries, the cerebral overlapping at the vertical meridian. One of the func- cortex was differentiated into discrete regions by the tions of the corpus callosum is to unify the perception anatomist Korbinian Brodmann and others using Kandel-Ch21_0496-0520.indd 502 20/01/21 2:43 PM  Chapter 21 / The Constructive Nature of Visual Processing  503 anatomical criteria. The criteria included the size, shape, and packing density of neurons in the cortical A Visual processing layers and the thickness and density of myelin. The functionally distinct cortical areas we have considered heretofore correspond only loosely to Brodmann’s Parietal cortex classification. The primary visual cortex (V1) is identi- Dorsal Thalamic nuclei: pathway cal to Brodmann’s area 17. In the extrastriate cortex, Pulvinar the secondary visual area (V2) corresponds to area 18. Lateral Beyond that, however, area 19 contains several func- geniculate nucleus tionally distinct areas that generally cannot be defined by anatomical criteria. The number of functionally discrete areas of SC visual cortex varies between species. Macaque Primary monkeys have more than 30 areas. Although not Inferotemporal visual cortex Ventral cortex all visual areas in humans have yet been identified, pathway the number is likely to be at least as great as in the macaque. If one includes oculomotor areas and pre- B Pupillary reflex and accommodation frontal areas contributing to visual memory, almost half of the cerebral cortex is involved with vision. Functional magnetic resonance imaging (fMRI) has made it possible to establish homologies between Pretectum the visual areas of the macaque and human brains (Figure 21–7). Based on pathway tracing studies in monkeys, we now appreciate that these areas are organized in functional streams (Figure 21–7B). The visual areas of cortex can be differentiated by the functional properties of their neurons. Studies of Accessory oculomotor nucleus Figure 21–6 Pathways for visual processing, pupillary reflex and accommodation, and control of eye position. Accommodation Ciliary ganglion A. Visual processing. The eye sends information first to tha- Pupillary reflex lamic nuclei, including the lateral geniculate nucleus and pulvi- nar, and from there to cortical areas. Cortical projections go forward from the primary visual cortex to areas in the parietal C Eye movement (horizontal) lobe (the dorsal pathway, which is concerned with visually guided movement) and areas in the temporal lobe (the ventral Posterior pathway, which is concerned with object recognition). The pulvi- parietal cortex nar also serves as a relay between cortical areas to supplement their direct connections. (Abbreviation: SC, superior colliculus). FEF B. Pupillary reflex and accommodation. Light signals are relayed through the midbrain pretectum, to preganglionic parasympa- Caudate thetic neurons in the accessory oculomotor (Edinger-Westphal) nucleus nucleus, and out through the parasympathetic outflow of the Substantia LGN oculomotor nerve to the ciliary ganglion. Postganglionic neu- nigra rons innervate the smooth muscle of the pupillary sphincter, as well as the muscles controlling the lens. SC C. Eye movement. Information from the retina is sent to the Primary superior colliculus (SC) directly along the optic nerve and indi- PPRF Abducens visual rectly through the geniculostriate pathway to cortical areas nucleus cortex (primary visual cortex, posterior parietal cortex, and frontal eye fields) that project back to the superior colliculus. The colliculus Abducens projects to the pons (PPRF), which then sends control signals nerve to oculomotor nuclei, including the abducens nucleus, which controls lateral movement of the eyes. (Abbreviations: FEF, frontal eye field; LGN, lateral geniculate nucleus; PPRF, para- median pontine reticular formation.) Kandel-Ch21_0496-0520.indd 503 20/01/21 2:43 PM  504  Part IV / Perception A Cortical visual areas in humans Normal Medial view Caudal view Lateral view Ventral view Inflated Occipital lobe (flattened) V1 V5/MT LO1 V2 V6 LO2 V3 IPSO pLOC V3A IPS1 FFA V3B IPS2 EBA hV4 VO1 PPA B Visual pathways in the macaque monkey Dorsal PMd MIP pathway FEF VIP LIP AIP MT/ PF PMv MST V3 MD PL V2 LGN V4 V1 TEO SC Feedforward Feedback IT Ventral pathway Kandel-Ch21_0496-0520.indd 504 20/01/21 2:43 PM  Chapter 21 / The Constructive Nature of Visual Processing  505 such functional properties have revealed that the vis- processing. The pulvinar in the thalamus serves as a ual areas are organized in two hierarchical pathways, relay between cortical areas (see Figure 21–7B). a ventral pathway involved in object recognition and The dorsal pathway courses through the parietal a dorsal pathway dedicated to the use of visual infor- cortex, a region that uses visual information to direct the mation for guiding movements. The ventral or object- movement of the eyes and limbs, that is, for visuomotor recognition pathway extends from the primary visual integration. The lateral intraparietal area, named for its cortex to the temporal lobe; it is described in detail in location in the intraparietal sulcus, is involved in repre- Chapter 24. The dorsal or movement-guidance path- senting points in space that are the targets of eye move- way connects the primary visual cortex with the pari- ments or reaching. Patients with lesions of parietal areas etal lobe and then with the frontal lobes. fail to attend to objects on one side of the body, a syndrome The pathways are interconnected so that informa- called unilateral neglect (see Figure 59–1 in Chapter 59). tion is shared. For example, movement information The ventral pathway extends into the temporal in the dorsal pathway can contribute to object rec- lobe. The inferior temporal cortex stores information ognition through kinematic cues. Information about about the shapes and identities of objects; one portion movements in space derived from areas in the dorsal represents faces, for damage to that region results in pathway is therefore important for the perception of the inability to recognize faces (prosopagnosia). object shape and is fed into the ventral pathway. The dorsal and ventral pathways each comprise a All connections between cortical areas are hierarchical series of areas that can be delineated by reciprocal—each area sends information back to the several criteria. First, at many relays, the array of inputs areas from which it receives input. These feedback forms a map of the visual hemifield. The boundaries of connections provide information about cognitive func- these maps can be used to demarcate the boundaries of tions, including spatial attention, stimulus expecta- visual areas. This is particularly useful at early levels tion, and emotional content, to earlier levels of visual of the pathway where the receptive fields of neurons Figure 21–7 Visual pathways in the cerebral cortex. codes faces, the parahippocampal place area (PPA) responds A. Functional magnetic resonance imaging shows areas of more strongly to places than to objects, the extrastriate body the human cerebral cortex involved in visual processing. The area (EBA) responds more strongly to body parts than objects, top row shows areas on the gyri and sulci of a normal view and V5/MT is involved in motion processing. Areas in the intra- of a brain; the middle row shows “inflated” views of the parietal sulcus (IPS1 and IPS2) are involved in control of spatial brain following a computational process that simulates inflat- attention and saccadic eye movements. (Images courtesy of V. ing the brain like a balloon so as to stretch out the “wrinkles” Piech, reproduced with permission.) of gyri and sulci into a smooth surface while minimizing local B. In the macaque monkey, V1 is located on the surface of distortions. Light and dark gray regions identify gyri and sulci, the occipital lobe and sends axons in two pathways. A dorsal respectively; the bottom row shows a two-dimensional repre- pathway courses through a number of areas in the parietal sentation of the occipital lobe (left) and a representation with lobe and into the frontal lobe and mediates attentional control less distortion by making a cut along the calcarine fissure. and visually guided movements. A ventral pathway projects Different approaches are required for demarcating different through V4 into areas of the inferior temporal cortex and medi- functional areas. Retinotopic areas, by definition, contain con- ates object recognition. In addition to feedforward pathways tinuous maps of visual space and are identified using stimuli extending from primary visual cortex into the temporal, pari- such as rotating spirals or expanding circles that sweep through etal, and frontal lobes (blue arrows), reciprocal or feedback visual space. Maps in adjacent cortical areas run in opposite pathways run in the opposite direction (red arrows). Feed- directions on the cortical surface and meet along boundaries forward and feedback can operate directly, between cortical of local mirror reversals. These mirror reversals can be used areas, or indirectly, via the thalamus, in particular the pulvinar, to identify area boundaries and thus demarcate each area. which acts as a relay between cortical areas. The subcortical These retinotopic areas, including early visual areas V1, V2, pathways involved include thalamic nuclei—the lateral genicu- and V3, and areas V3A, V3B, V6, hV4, VO1, LO1, LO2, and V5/ late nucleus (LGN), pulvinar nucleus (PL), and mediodorsal MT, share boundaries in pairs; these boundaries converge (at nucleus (MD)—and the superior colliculus (SC). (Abbreviations: the representation of the fovea) at the occipital pole. A differ- AIP, anterior intraparietal area; FEF, frontal eye field; IT, infe- ent approach, identifying loci of attention, is used to map areas rior temporal cortex; LIP, lateral intraparietal area; MIP, medial IPS1 and IPS2. Yet further sets of approaches or responsive- intraparietal area; MT, middle temporal area; PF, prefrontal ness to specific attributes or classes of objects (such as faces) cortex; PMd, dorsal premotor cortex; PMv, ventral premotor are used for less strictly retinotopic areas. Functional specificity cortex; TEO, posterior division of area IT; V1, primary visual has been demonstrated for a number of visual areas: VO1 is cortex, Brodmann’s area 17; V2, secondary visual area, implicated in color processing, the lateral occipital complex Brodmann’s area 18; V3, V4, third and fourth visual areas; VIP, (LO2, pLOC) codes object shape, fusiform face area (FFA) ventral intraparietal area.) Kandel-Ch21_0496-0520.indd 505 20/01/21 2:43 PM  506  Part IV / Perception are small and visuotopic maps are precisely organ- to record from single neurons in the eye, H. Keffer ized (see the next section for the definition of receptive Hartline applied the concept of the receptive field in field). At higher levels, however, the receptive fields his study of the retina of the horseshoe crab, Limulus: become larger, the maps less precise, and visuotopic “The region of the retina which must be illuminated organization is therefore a less reliable basis to deline- in order to obtain a response in any given fiber... is ate the boundaries of an area. termed the receptive field of that fiber.” In the visual Another means to differentiate one area from system, a neuron’s receptive field represents a small another, as shown by experiments in monkeys, window on the visual field (Figure 21–8). depends upon the distinctive functional properties But responses to only one spot of light yielded a exhibited by the neurons in each area. The clearest limited understanding of a cell’s receptive field. Using example of this is an area in the dorsal pathway, the two small spots of light, both Hartline and Stephen middle temporal area (MT or V5), which contains neu- Kuffler, who studied the mammalian retina, found an rons with a strong selectivity for the direction of move- inhibitory surround or lateral inhibitory region in the ment across their receptive fields. Consistent with the receptive field. In 1953, Kuffler observed that “not only idea that the middle temporal area is involved in the the areas from which responses can actually be set up analysis of motion, lesions of this area produce deficits by retinal illumination may be included in a definition in the ability to track moving objects. of the receptive field but also all areas which show a A classical view of the organization of visual corti- functional connection, by an inhibitory or excitatory cal areas is a hierarchical one, where the areas at the effect on a ganglion cell.” Kuffler thus demonstrated bottom of the hierarchy, such as V1 and V2, represent that the receptive fields of retinal ganglion cells have the visual primitives of orientation, direction of move- functionally distinct subareas. These receptive fields ment, depth, and color. In this view, the top of the have a center-surround organization and fall into one ventral pathway’s hierarchy would represent whole of two categories: on-center and off-center. Later work objects, with the areas in between representing inter- demonstrated that neurons in the LGN have similar mediate level vision. This idea of “complexification” receptive fields. along the hierarchy suggests a mapping between the The on-center cells fire when a spot of light is turned levels of visual perception and stages in the sequence on within a circular central region. Off-center cells fire of cortical areas. But more recent findings indicate a when a spot of light in the center of their receptive field more complex story, where even the primary visual is turned off. The surrounding annular region has the cortex plays a role in intermediate-level vision, and opposite sign. For on-center cells, a light stimulus any- neurons in the higher areas may process informa- where in the annulus surrounding the center produces tion on components of objects. Moreover, as shown in a response when the light is turned off, a response Figure 21–7, one also has to take into account the fact termed on-center, off-surround. The center and surround that there is a powerful reverse flow of information, areas are mutually inhibitory (Figure 21–9). When both or feedback, from the “higher” to the “lower” cortical center and surround are illuminated with diffuse light, areas. As will be described in Chapter 23, this reverse there is little or no response. Conversely, a light–dark direction of information contains higher order “top- boundary across the receptive field produces a brisk down” cognitive influences including attention, object response. Because these neurons are most sensitive to expectation, perceptual task, perceptual learning, and borders and contours—to differences in illumination efference copy. Top-down influences may play a role in as opposed to uniform surfaces—they encode informa- scene segmentation, object relationships, and percep- tion about contrast in the visual field. tion of object details, as well as object recognition itself. The size on the retina of a receptive field varies both according to the field’s eccentricity—its posi- tion relative to the fovea, the central part of the retina The Receptive Fields of Neurons at Successive where visual acuity is highest—and the position of Relays in the Visual Pathway Provide Clues to neurons along the visual pathway. Receptive fields How the Brain Analyzes Visual Form with the same eccentricity are relatively small at early levels in visual processing and become progressively In 1906, Charles Sherrington coined the term receptive larger at later levels. The size of the receptive field field in his analysis of the scratch withdrawal reflex: is expressed in terms of degrees of visual angle; the “The whole collection of points of skin surface from entire visual field covers nearly 180° (Figure 21–10A). which the scratch-reflex can be elicited is termed the In early relays of visual processing, the receptive receptive field of that reflex.” When it became possible fields near the fovea are the smallest. The receptive Kandel-Ch21_0496-0520.indd 506 20/01/21 2:43 PM  Chapter 21 / The Constructive Nature of Visual Processing  507 A Receptive fields on the retina Receptive field in the periphery Receptive field near the fovea B Receptive field of a retinal ganglion cell Retinal Horizontal, bipolar Photoreceptors contributing to Center-surround structure of ganglion cell and amacrine cells ganglion cell receptive field ganglion cell receptive field On area (center) Off area (surround) Light Figure 21–8 Receptive fields of retinal ganglion cells in rela- whereas a cell farther from the fovea receives input from many tion to photoreceptors. more receptors covering a larger area (see Figure 21–10). A. The number of photoreceptors contributing to the receptive B. Light passes through nerve cell layers to reach the photore- field of a retinal ganglion cell varies depending on the loca- ceptors at the back of the retina. Signals from the photorecep- tion of the receptive field on the retina. A cell near the fovea tors are then transmitted by neurons in the outer and inner receives input from fewer receptors covering a smaller area, nuclear layers to a retinal ganglion cell. fields for retinal ganglion cells that monitor portions analyzed by the brain. For example, the change in of the fovea subtend approximately 0.1°, whereas receptive-field structure that occurs between the LGN those in the visual periphery can be a couple of orders and cerebral cortex reveals an important mechanism of magnitude larger. in the brain’s analysis of visual form. The key prop- The amount of cortex dedicated to a degree of erty of the form pathway is selectivity for the orienta- visual space changes with eccentricity. More area of tion of contours in the visual field. This is an emergent cortex is dedicated to the central part of the visual property of signal processing in primary visual cortex; field, where the receptive fields are smallest and it is not a property of the cortical input but is generated the visual system has the greatest spatial resolution within the cortex itself. (Figure 21–10C). Whereas retinal ganglion cells and neurons in Receptive-field properties change from relay to the LGN have concentric center-surround receptive relay along a visual pathway. By determining these fields, those in the cortex, although equally sensitive properties, one can assay the function of each relay to contrast, also analyze contours. David Hubel and nucleus and how visual information is progressively Torsten Wiesel discovered this characteristic in 1958 Kandel-Ch21_0496-0520.indd 507 20/01/21 2:44 PM  508  Part IV / Perception while studying what visual stimuli provoked activ- ity in neurons in the primary visual cortex. While showing an anesthetized animal slides containing a variety of images, they recorded extracellularly from individual neurons in the visual cortex. As Lateral geniculate Retinal nucleus neuron they switched from one slide to another, they found ganglion cell a neuron that produced a brisk train of action poten- tials. The cell was responding not to the image on the slide but to the edge of the slide as it was moved into position. The Visual Cortex Is Organized Into Columns Response of Specialized Neurons Stimulus The dominant feature of the functional organization of the primary visual cortex is the visuotopic organiza- tion of its cells: the visual field is systematically repre- Off area (surround) sented across the surface of the cortex (Figure 21–11A). In addition, cells in the primary visual cortex with similar functional properties are located close together On area in columns that extend from the surface of the cortex to (center) the white matter. The columns are concerned with the functional properties that are analyzed in any given cortical area and thus reflect the functional role of that area in vision. The properties that are developed in the primary visual cortex include orientation specificity and the integration of inputs from the two eyes, which is measured as the relative strength of input from each eye, or ocular dominance. Ocular-dominance columns reflect the segrega- tion of thalamocortical inputs arriving from different layers of the LGN. Alternating layers of this nucleus receive input from retinal ganglion cells located in either the ipsilateral or contralateral retina (Figure 21–12). This segregation is maintained in the inputs from the LGN to the primary visual cortex, produc- ing the alternating left-eye and right-eye ocular domi- nance bands (Figure 21–11B). Cells with similar orientation preferences are also grouped into columns. Across the cortical surface, there is a regular clockwise and counterclockwise cycling of orientation preference, with the full 180° cycle repeat- Figure 21–9 Receptive fields of neurons at early relays of ing every 750 μm (Figure 21–11C). Likewise, the left- visual pathways. A circular symmetric receptive field with and right-eye dominance columns alternate with a mutually antagonistic center and surround is characteristic of reti- nal ganglion cells and neurons in the lateral geniculate nucleus periodicity of 750 to 1,000 μm. One full cycle of orienta- of the thalamus. The center can respond to the turning on or tion columns, or a full pair of left- and right-eye domi- turning off of a spot of light (yellow) depending on whether the nance columns, is called a hypercolumn. The orientation receptived field belongs to an “on-center” or “off-center” class, and ocular dominance columns at each point on the respectively. The surround has the opposite response. Outside cortical surface are locally roughly orthogonal to each the surround, there is no response to light, thus defining the receptive field boundary. The response is weak when light covers other. Thus, a cortical patch one hypercolumn in extent both the center and surround, so these neurons respond contains all possible combinations of orientation pref- optimally to contrast (a light–dark boundary) in the visual field. erence and left- and right-eye dominance. Kandel-Ch21_0496-0520.indd 508 20/01/21 2:44 PM  Chapter 21 / The Constructive Nature of Visual Processing  509 A Map of retinal eccentricity Both types of columns were first mapped by record- ing the responses of neurons at closely spaced electrode penetrations in the cortex. The ocular-dominance col- umns were also identified by making lesions or tracer injections in individual layers of the LGN. More recently, a technique known as optical imaging has enabled researchers to visualize a surface representation of the 10 20 30 40 50 60 orientation and ocular dominance columns in living ani- mals. Developed for studies of cortical organization by Amiram Grinvald, this technique visualizes changes in surface reflectance associated with the metabolic require- ments of active groups of neurons, known as intrinsic- signal optical imaging, or changes in fluorescence of voltage-sensitive dyes. Intrinsic-signal imaging depends on activity-associated changes in local blood flow and B Receptive field size varies systematically with eccentricity alterations in the oxidative state of hemoglobin and other intrinsic chromophores. These techniques are also now Receptive field diameter (°) 2 being complemented with imaging at cellular resolution using genetically encoded markers of neural activity. An experimenter can visualize the distribution of 1 cells with left or right ocular dominance, for example, by subtracting the image obtained while stimulating one eye from that acquired while stimulating the other. 10 20 30 When viewed in a plane tangential to the cortical sur- Eccentricity (°) face, the ocular dominance columns appear as alternat- ing left- and right-eye stripes, each approximately 750 μm in width (Figure 21–11B). C Cortical magnification varies with eccentricity The cycles of orientation columns form various struc- 10 tures, from parallel stripes to pinwheels. Sharp jumps in 20 orientation preference occur at the pinwheel centers and 30 V1 40 “fractures” in the orientation map (Figure 21–11C). 50 60 Embedded within the orientation and ocular- dominance columns are clusters of neurons that have poor orientation selectivity but strong color prefer- ences. These units of specialization, located within the superficial layers, were revealed by a histochemi- cal label for the enzyme cytochrome oxidase, which is distributed in a regular patchy pattern of blobs and interblobs. In the primary visual cortex, these blobs are a few hundred micrometers in diameter and 750 μm Figure 21–10 Receptive field size, eccentricity, retinotopic organization, and magnification factor. The color code refers apart (Figure 21–11D). The blobs correspond to clus- to position in visual space or on the retina. ters of color-selective neurons. Because they are rich in A. The distance of a receptive field from the fovea is referred to cells with color selectivity and poor in cells with orien- as the eccentricity of the receptive field. tation selectivity, the blobs are specialized to provide B. Receptive field size varies with distance from the fovea. The information about surfaces rather than edges. smallest fields lie in the center of gaze, the fovea, where the In area V2, thick and thin dark stripes separated by visual resolution is highest; fields become progressively larger pale stripes are evident with cytochrome oxidase labe- with distance from the fovea. ling (Figure 21–11D). The thick stripes contain neurons C. The amount of cortical area dedicated to inputs from within selective for direction of movement and for binocular each degree of visual space, known as the magnification factor, disparity as well as cells that are responsive to illusory also varies with eccentricity. The central part of the visual field commands the largest area of cortex. For example, in area V1, contours and global disparity cues. The thin stripes more area is dedicated to the central 10° of visual space than to hold cells specialized for color. The pale stripes contain all the rest. The map of V1 shows the cortical sheet unfolded. orientation-selective neurons. Kandel-Ch21_0496-0520.indd 509 20/01/21 2:44 PM  510  Part IV / Perception A Visuotopic map V2 Pattern of excitation V1 in response Stimulus to striped stimulus V2 V1 B Ocular dominance columns V2 V1 Left eye Right eye C Orientation columns Orientation preference Thin D Blobs, interblobs (V1), and stripes (V2) stripe Stripes Thick stripe Blobs Kandel-Ch21_0496-0520.indd 510 20/01/21 2:44 PM  Chapter 21 / The Constructive Nature of Visual Processing  511 Figure 21–11 (Opposite) Functional architecture of the intersect the border between areas V1 and V2, the representa- primary visual cortex. (Courtesy of M. Kinoshita and A. Das, tion of the vertical meridian, at right angles. reproduced with permission.) C. Some columns contain cells with similar selectivity for the A. The surface of the primary visual cortex is functionally organ- orientation of stimuli. The different colors indicate the orienta- ized as a map of the visual field. The elevations and azimuths tion preference of the columns. The orientation columns in of visual space are organized in a regular grid that is distorted surface view are best described as pinwheels surrounding because of variation in the magnification factor (see Figure 21–10). singularities of sudden changes in orientation (the center of the The grid is visible here in the dark stripes (visualized with intrinsic- pinwheel). The scale bar represents 1 mm. (Surface image of signal optical imaging), which reflect the pattern of neurons that orientation columns on the left courtesy of G. Blasdel, repro- responded to a series of vertical candy stripes. Within this surface duced with permission.) map, one finds repeated superimposed cycles of functionally spe- D. Patterns of blobs in V1 and stripes in V2 represent other cific columns of cells, as illustrated in B, C, and D. modules of functional organization. These patterns are visual- B. The dark and light stripes represent the surface view of ized with cytochrome oxidase. the left and right ocular dominance columns. These stripes Binocular zone of right hemiretina Monocular zone Contralateral Ipsilateral Optic tract Dorsal Optic Optic nerve chiasm IVA Optic tract IVB Ventral IVCα Lateral 1 2 3 4 5 6 IVCβ geniculate nucleus Parvocellular pathway (P channel) Magnocellular pathway (M channel) Figure 21–12 Projections from the lateral geniculate layers: The contralateral eye projects to layers 1, 4, and 6, nucleus to the visual cortex. The lateral geniculate nucleus whereas the ipsilateral eye sends input to layers 2, 3, and 5. in each hemisphere receives input from the temporal retina of Neurons from these geniculate layers then project to different the ipsilateral eye and the nasal retina of the contralateral eye. layers of cortex. The parvocellular geniculate neurons project The nucleus is a layered structure comprising four parvocellular to layer IVCβ, the magnocellular ones project to layer IVCα, and layers (layers 3 to 6) and two magnocellular layers (layers 1 and 2). the koniocellular ones project to “blobs” in the upper cortical Each is paired with an intercalated koniocellular layer. (These layers (see Figures 21–14 and 21–15). In addition, the afferents layers are represented here by the gaps separating the primary from the ipsilateral and contralateral layers of the lateral genicu- layers. They are unlabeled to avoid clutter. See Figure 21–14.) late nucleus are segregated into alternating ocular-dominance The inputs from the two eyes terminate in different geniculate columns. Kandel-Ch21_0496-0520.indd 511 20/01/21 2:44 PM  512  Part IV / Perception For every visual attribute to be analyzed at each Columnar organization confers several advantages. position in the visual field, there must be adequate til- It minimizes the distance required for neurons with ing, or coverage, of neurons with different functional similar functional properties to communicate with one properties. As one moves in any direction across the another and allows them to share inputs from discrete cortical surface, the progression of the visuotopic loca- pathways that convey information about particular sen- tion of receptive fields is gradual, whereas the cycling sory attributes. This efficient connectivity economizes of columns occurs more rapidly. Any given position in on the use of brain volume and maximizes processing the visual field can therefore be analyzed adequately speed. The clustering of neurons into functional groups, in terms of the orientation of contours, the color and as in the columns of the cortex, allows the brain to mini- direction of movement of objects, and stereoscopic mize the number of neurons required for analyzing depth by a single computational module. The small different attributes. If all neurons were tuned for every segment of visual cortex that comprises such a module attribute, the resultant combinatorial explosion would represents all possible values of all the columnar sys- require a prohibitive number of neurons. tems (Figure 21–13). The columnar systems serve as the substrate for two fundamental types of connectivity along the vis- Intrinsic Cortical Circuits Transform ual pathway. Serial processing occurs in the successive Neural Information connections between cortical areas, connections that run from the back of the brain forward. At the same Each area of the visual cortex transforms information time, parallel processing occurs simultaneously in sub- gathered by the eyes and processed at earlier synap- sets of fibers that process different submodalities such tic relays into a signal that represents the visual scene. as form, color, and movement, continuing the neural This transformation is accomplished by local circuits processing strategy started in the retina. comprising both excitatory and inhibitory neurons. Many areas of visual cortex reflect this arrange- The principal input to the primary visual cortex ment; for example, functionally specific cells in V1 comes from three parallel pathways that originate communicate with cells of the same specificity in V2. in the parvocellular, magnocellular, and the blue/ These pathways are not absolutely segregated, how- yellow channels of koniocellular layers of the LGN ever, for there is some mixing of information between (see Figure 21–12). Neurons in the parvocellular lay- different visual attributes (Figure 21–14). ers project to cortical layers IVCβ and 6, those in the Figure 21–13 A cortical computational module. A chunk of cortical tissue roughly 1 mm square contains an orientation hypercolumn (a full cycle of Blobs orientation columns), one cycle of left- and right- eye ocular-dominance columns, and blobs and interblobs. This module would presumably contain all of the functional and anatomical cell types of primary visual cortex, which would be repeated hundreds of times to cover the visual field. (Adapted from Hubel 1988.) Orientation columns Ocular dominance columns: Left eye Right eye Orientation preference Kandel-Ch21_0496-0520.indd 512 20/01/21 2:44 PM  Chapter 21 / The Constructive Nature of Visual Processing  513 Dorsal (parietal) pathway MT Ventral (temporal) Color pathway Depth Thick stripe Thin stripe Direction Interstripe Orientation From Complex form V4 interblobs II V2 From III blobs IVB IVCα IVCβ Parvocellular V Koniocellular VI Midget Parvo ganglion pathway cells Dorsal V1 (parietal) V2 Bistratified Konio pathway MT Magnocellular ganglion pathway cells LGN LGN Parasol Magno ganglion pathway V1 cells V4 Retina Retina Ventral V2 (temporal) pathway Figure 21–14 Parallel processing in visual pathways. The segregated, however, and there is substantial interconnection ventral stream i