The Visual Cortex: Two Streams, One for Seeing and One for Doing

Introduction

A woman sits at a table. In front of her is a vertical slot cut into a disc, like the slot of a postbox. A researcher asks her a simple question: is the slot vertical, horizontal, or tilted?

She cannot answer. She guesses. She is wrong about as often as she is right.

Then the researcher hands her a card and asks her to post it through the slot. She reaches out, and on the way her wrist rotates to exactly the angle needed, and the card slides in cleanly [1].

She cannot see the orientation of the slot. Her hand can.

The patient is known in the literature as DF. In 1988 she was poisoned by carbon monoxide from a faulty water heater, and the damage settled into a particular region on the underside of both occipital lobes [2]. What she lost was the ability to perceive the shape and orientation of things. What she kept was the ability to act on them.

Somewhere else, in a different clinic, there are patients with the opposite problem. Show them an object and they describe it perfectly. Size, shape, angle, all correct. Ask them to pick it up and their hand goes to the wrong place, or arrives with the fingers set to the wrong width, and they grope for it like someone reaching in the dark [3]. The condition is called optic ataxia, and it follows damage to a region higher up and further back, in the parietal lobe.

Two patients. Two lesions. Two deficits that are mirror images of each other.

That pairing is the reason anyone studies the visual cortex with more than anatomical interest. It suggests something genuinely strange about how vision works: that the seeing you experience and the seeing that steers your hands are not the same process running at different volumes. They are different processes, running in different tissue, and one can be destroyed while the other carries on.

This article is about how the visual cortex is built, what its parts do, and how far that two-stream story can actually be pushed. The short version is that it can be pushed further than most explanations go, and not as far as most explanations claim.

The Route Every Image Takes

Before any of the interesting arguments, there is a piece of plumbing worth getting right, because nearly everyone gets it wrong in the same way.

Light hits the retina. The retina does real processing of its own, then sends signals down the optic nerve. The two optic nerves meet at a crossing point called the optic chiasm, and here is where the common mistake happens.

The chiasm does not send the left eye to the right hemisphere. It sends the left half of the visual field to the right hemisphere, and the right half to the left hemisphere. Both eyes contribute to both. Fibres from the nasal half of each retina cross over; fibres from the temporal half stay on their own side. The result is that each hemisphere gets a complete picture of the opposite half of the world, assembled from both eyes.

That detail is not pedantry. It is the reason a stroke in one occipital lobe does not blind one eye. It knocks out one half of the visual field in both eyes, a pattern called homonymous hemianopia, and a patient with it will bump into door frames on one side while insisting both eyes work fine.

After the chiasm, the signal reaches the lateral geniculate nucleus, a small layered structure buried in the thalamus, the relay station almost every sense has to pass through. From there it travels back to the very rear of the brain, into a fold called the calcarine sulcus, and arrives at the primary visual cortex. V1. Also called the striate cortex, because a stripe of pale myelinated fibres is visible in it with the naked eye.

V1 is where cortical vision starts. It is not where seeing happens. That distinction matters and it will keep mattering.

If you want a sense of how fast this is, the whole feedforward sweep from retina to high-level object recognition takes a fraction of a second. Recognition of a complex scene can be underway within about 150 milliseconds, which is less time than it takes to blink.

What a Single Neuron in V1 Actually Wants

In the late 1950s, two researchers at Johns Hopkins were trying to find out what makes a cell in the visual cortex fire. David Hubel and Torsten Wiesel had an anaesthetised cat, a microelectrode in its striate cortex, and a slide projector aimed at a screen.

They shone spots of light. Almost nothing happened. The cells sat there, indifferent.

The breakthrough came, as the story is usually told, by accident. While changing a glass slide, the edge of the slide swept across the projector beam and cast a moving line on the screen. A cell that had ignored everything for hours suddenly fired hard.

It was not the light. It was the edge.

What Hubel and Wiesel had found is that neurons in V1 do not report brightness at a point. They report the presence of an edge at a particular orientation, in a particular small patch of the visual field, and often moving in a particular direction [4]. A cell that fires enthusiastically for a bar tilted at 45 degrees will barely respond to the same bar tilted at 90.

Their 1962 paper laid out the architecture that followed from this [5]. Simple cells respond to an edge at one orientation in one exact position. Complex cells respond to the same orientation anywhere within a larger region, which makes them tolerant of small movements. And the cells are not scattered at random. Drive an electrode straight down through the cortex and every cell you pass prefers roughly the same orientation. Move sideways a fraction of a millimetre and the preferred orientation shifts slightly. The cortex is organised into columns, and the columns step through orientations in an orderly progression.

Hubel and Wiesel shared the Nobel Prize in 1981 for this work. It is worth being clear about what kind of evidence it is: single-electrode recordings in individual anaesthetised cats, done one neuron at a time over years. Not a large cohort study. The finding earned its status by replicating in species after species and by holding up when the technology changed completely.

And it did hold up. In 2005 a group using two-photon calcium imaging watched hundreds of individual neurons at once in living cortex and saw the orientation map directly, cell by cell, exactly as the electrode studies had predicted from single points [6].

The deeper idea underneath all this is that vision is not a photograph being passed along. At the very first cortical stage, the image has already been thrown away and replaced with a description: edges here, at these angles, moving this way. Everything that follows works on the description, never on the picture. If the basic machinery of how one cell talks to the next is unfamiliar, the mechanics of neural signalling are worth a detour before going further.

The Map Inside the Map

V1 is laid out as a map of the visual field. Neighbouring points in what you see are handled by neighbouring points of cortex. This is called retinotopic organisation, and it is one of the most reliable facts in the whole of neuroscience.

The map is not drawn to scale. The centre of gaze, the fovea, occupies a wildly disproportionate share of the cortical surface relative to its tiny area on the retina. Peripheral vision is squeezed into comparatively little tissue. This is called cortical magnification, and it is why fine detail collapses the moment something moves away from where you are looking directly. Published estimates of the exact ratio vary widely depending on method, so the honest statement is the direction of the effect rather than a number.

Two very different eras of evidence established this map.

The first was war. In 1909, a Japanese physician named Tatsuji Inouye studied soldiers wounded in the Russo-Japanese War. Rifle bullets travelling through the skull left narrow, traceable paths, and the blind patches those soldiers were left with corresponded to the entry and exit points in a systematic way. Inouye worked out a rough cortical map from the geometry. During the First World War, Gordon Holmes did the same on a much larger scale with British casualties, and published a detailed account of how occipital wounds map onto visual field defects [7]. The maps neuroscience still teaches were first sketched from gunshot wounds.

The second era was imaging. In 1994 a group used functional MRI to map human V1 non-invasively for the first time, using a stimulus that swept around the visual field like a clock hand [8]. A year later, another team pushed the technique far enough to find the borders between multiple distinct visual areas in living people [9], and by 1997 the precision of the method itself had been quantified [10]. A modern synthesis of human visual field maps collects a dozen such areas [11].

One caution about numbers here. You will often read that the brain contains more than thirty visual areas. That figure comes from a famous 1991 paper mapping the cortical hierarchy in the macaque monkey [12]. It is a monkey map. The human count is not settled, and quoting the macaque figure as a human fact is one of the most common errors in popular writing on this subject.

A Job List, Not an Alphabet

Past V1, the cortex divides into further areas with names that read like a filing system: V2, V3, V4, V5. The names are unhelpful. What matters is what each contributes and what goes missing when it fails.

V2 takes V1's edges and starts assembling them into contours, including illusory contours, the edges you see in a shape that is not physically drawn. V3 is involved in dynamic form. V4 is heavily involved in colour and in more complex shape. V5, usually called MT, is the motion area.

A landmark 1988 review argued that form, colour, movement and depth are carried by partly separate channels running in parallel rather than by one general-purpose stream [13]. Three years later, imaging in humans demonstrated functional specialisation directly, showing colour and motion stimuli activating distinct cortical territory [14].

The V5 row deserves its own section, because losing that one area produces the most unsettling clinical description in this whole subject.

When Motion Dies

In 1983 three researchers in Munich described a patient known as LM [15]. She had suffered damage to motion-sensitive cortex on both sides.

Her acuity was fine. Colour vision fine. She could read, recognise faces, and identify objects without difficulty. What she had lost was motion.

Not blurred motion. Not slow motion. Motion as a perceptual category simply was not available to her. She described pouring tea as watching a frozen column of liquid, like a glacier, with no sense of the level rising, so the cup overflowed before she registered that it was filling. People in a room did not move across it. They were in one place, and then they were in another place, with nothing in between. Crossing a road became dangerous, because a car that was far away was suddenly close, and she had no way to judge the approach.

This is a single case. One patient. That is exactly why it carries weight: the damage was bilateral and specific, and the deficit was equally specific. Motion vision turned out to be a separable thing that can be removed on its own.

The animal work fills in the mechanism, and it is unusually direct. Lesions restricted to area MT in monkeys selectively impair motion perception while leaving other visual judgements intact [16]. Recordings showed that the firing of individual MT neurons tracks the animal's perceptual decisions closely enough to predict, trial by trial, what the animal will report [17].

Then came the experiment that turns correlation into cause. In 1990, researchers passed tiny electrical currents into clusters of MT neurons tuned to a particular direction while a monkey watched an ambiguous display of moving dots and reported which way they drifted. The stimulation biased the animal's reports toward the direction those neurons preferred [18].

They changed what the monkey said it saw by injecting current into a few dozen cells.

There are not many results in neuroscience that clean. A review of MT's structure and function collects the rest of the evidence [19], but the microstimulation study is the one worth remembering, because it establishes that this small patch of cortex is not merely correlated with the experience of motion. It participates in producing it.

Two Streams

Now the argument the article is really about.

By the early 1980s it was clear that visual areas beyond V1 were not arranged in a single chain. Two broad routes could be traced. One runs downward and forward from V1 into the temporal lobe. The other runs upward and back into the parietal lobe.

In 1983, Mortimer Mishkin, Leslie Ungerleider and Kathleen Macko proposed what these two routes are for [20]. The temporal route, they argued, identifies objects. The parietal route locates them in space. Object vision and spatial vision. What and where.

That version is the one that made it into textbooks and it is the one most articles still repeat. It is also not the version that the patient evidence supports.

In 1992, Melvyn Goodale and A. David Milner published a reframing [21]. The difference between the streams, they argued, is not what information they carry but what the information is for. The temporal stream builds your conscious perception of the world, the enduring description you can report, remember and reason about. The parietal stream computes something quite different: the moment-by-moment coordinate transformations needed to actually move a hand to an object. Not what and where. What and how.

The distinction sounds subtle and it is not. On the what/where account, both streams are producing knowledge, just about different properties. On the what/how account, only one of them is producing knowledge at all. The other is running a control system whose output is a movement, not a thought, and whose contents you have no direct access to.

The clean way to say it: one stream is for seeing, the other is for doing.

Both streams start in V1, so this is a fork rather than two independent systems running from the eye. Both also feed forward eventually into the prefrontal cortex, where goals and plans are assembled, which is part of why the strict separation gets complicated further along.

The ventral stream is where the brain's specialist recognition modules sit. A region of lateral occipital cortex responds to objects as objects, regardless of whether they are defined by shading, motion or texture [22], and it holds up across changes in viewpoint and lighting [23]. Further along there is a patch that responds far more to faces than to other objects [24]. And in literate people there is a region on the left that responds to written words specifically, which is remarkable given that writing is far too recent for evolution to have built anything for it [25]. That region has a story of its own worth reading separately.

The dorsal stream, meanwhile, lights up when people actually reach for and grasp things. Imaging studies that separate grasping from merely looking find activation in parietal areas and not in the ventral object areas [26].

The Patient Who Could Not See the Slot

Return to DF, because the whole framework rests on her.

Carbon monoxide poisoning damaged the lateral occipital region on both sides, sparing V1 substantially. What resulted is called visual form agnosia. Her problem is not acuity, not colour, not brightness. She cannot derive shape from what she sees. Shown a drawing of an apple she cannot say what it is; handed a real apple she identifies it instantly by touch. Asked to copy a line drawing she produces something unrecognisable; asked to draw an apple from memory she draws a perfectly good apple [2]. The knowledge is intact. The visual route to it is severed.

Then the slot experiment. Asked to report the orientation of the slot, she performs near chance. Asked to post a card through it, her hand rotates correctly during the reach [1]. Asked to indicate the width of a block between finger and thumb, she is inaccurate. Asked to pick the block up, her grip aperture scales to its actual size before her fingers touch it.

The same visual property, orientation, is unavailable to one system and available to another, in the same brain, at the same moment.

It is worth stating plainly that this is one patient. n equals 1. A single case, however striking, cannot carry a theory of cortical organisation on its own. What makes DF persuasive is not her alone. It is that somebody else, with damage in the other place, shows the exact inverse.

The Mirror Image

Optic ataxia follows damage to the posterior parietal cortex, usually on both sides. The name is old and slightly misleading, since these patients are not ataxic in the cerebellar sense.

What they have is a specific failure of visually guided reaching. They see the object. They recognise it, describe it, tell you its shape and size and orientation without difficulty. Then they reach for it and the hand goes wide, or arrives with the grip set to the wrong size, or closes on empty air. The deficit is worst for objects in peripheral vision and improves when the patient can look directly at the target. A detailed 1988 study of a patient series established the pattern and argued it reflects disruption of visuomotor mechanisms specifically rather than of vision or of movement [3].

Put the two side by side.

This is what a double dissociation looks like, and it is the strongest form of evidence classical neuropsychology can produce. If one lesion removes A and spares B, that could mean A is simply harder. If a second lesion removes B and spares A, difficulty cannot explain it. The two abilities must depend on at least partly separate machinery.

Goodale and Milner restated and defended the framework in 2008, sixteen years after the original proposal [27]. By then it had become one of the most cited ideas in visual neuroscience.

It had also started to come apart.

The Illusion That Fools the Eye but Not the Hand

Before the trouble, one more piece of evidence, because it is the most quoted result in the field and it deserves careful handling.

If perception and action really run on separate machinery, an illusion that fools one might leave the other untouched. In 1995 a group tested this with the Ebbinghaus illusion, in which a disc surrounded by large circles looks smaller than an identical disc surrounded by small ones. They made the discs into solid, graspable objects and measured how wide people opened their fingers as they reached [28].

The verbal reports followed the illusion. The grip apertures tracked the real physical size much more closely.

The eye was fooled. The hand was not. As a demonstration it is almost too good, and it has been repeated in popular accounts ever since as settled fact.

It is not settled. In 2000 another group re-ran the experiment with changes to the method, particularly to how the perceptual comparison was made and how the visual display was arranged, and found no dissociation at all: grasping was influenced by the illusion to about the same degree as perception [29]. They argued the original result came from a mismatch between the perceptual and motor tasks rather than from separate systems. The exchange ran for years, with a 2008 summary maintaining that the data are consistent once methods are equated and no dissociation survives [30]. Others pushed back, arguing that differences in attentional demand cannot account for the effect either [31].

Anyone who tells you flatly that grasping is immune to visual illusions is reporting half of a live argument. That is the honest position, and it is worth holding, because the illusion result is not what the model rests on anyway. The patients are.

The Model Under Fire

Here is what most explanations of the visual cortex never get to, and it is the most interesting part.

Start with the awkward finding. In 2018 a team tested DF's reaching more carefully than anyone had before, particularly for targets in peripheral vision and for corrections made mid-reach. Her performance was impaired. The paper is titled, without much ambiguity, "Optic ataxia with visual form agnosia" [32].

The founding patient of the two visual systems hypothesis has features of both syndromes.

This did not come out of nowhere. Earlier work had already found that DF's visuomotor performance based on peripheral vision was impaired [33], and a 2014 study asked directly whether her dorsal stream was intact and concluded that it was not, pointing to thinning of grey matter in dorsal regions in both hemispheres [34]. Carbon monoxide poisoning is diffuse. It was never likely to have carved out one stream with surgical tidiness.

Meanwhile the same doubt was arriving from the other direction. If optic ataxia is a pure dorsal deficit, perception in those patients should be untouched. A 2011 analysis found that deficits of perception and action in optic ataxia are correlated rather than dissociated [35]. Earlier work had already documented interaction between the streams in these patients [36], and further studies argued the same [37] and reassessed what optic ataxia tells us about dorsal function [38]. A broader challenge asked the question in its title: do we have independent visual streams for perception and action [39].

What happened next is the part that reflects well on the field. Goodale and Milner's own group published the reassessment. A 2014 paper by Robert Whitwell with both original authors reviews the new challenges to the two visual systems hypothesis and the insights from DF, and revises the account rather than defending it unchanged [40].

There is also a serious rival framework. In 2011 a group including Mortimer Mishkin, co-author of the original what/where proposal, argued that the dorsal stream is not one pathway but at least three, running to the prefrontal cortex, the premotor cortex and the medial temporal lobe, and supporting spatial working memory, visually guided action and finding your way through space respectively [41]. Others have questioned whether the what and where labels are useful descriptions of the anatomy at all [42].

So where does that leave the two-stream story?

Intact in its careful form, and wrong in its popular one. The dissociation is real: brain damage can take away your ability to see the shape of a thing while leaving your ability to grasp it, and it can do the reverse. That is a genuine and surprising fact about how vision is organised. What does not survive is the stronger claim, the one in most diagrams, that these are two sealed systems running side by side without speaking. They interact constantly. They share inputs. They can be damaged together, and usually are.

The right sentence is that the two streams can come apart. Not that they are apart.

Seeing Without Knowing

There is a further route through the visual system that the two-stream story does not cover at all, and it produces the strangest phenomenon in this whole article.

In 1974 a team reported on a patient known as DB, who had undergone surgical removal of part of his occipital cortex [43]. He was blind in the corresponding part of his visual field. He said so. Asked whether he could see anything there, he said no, consistently and sincerely.

Then they asked him to guess.

Where is the light: left or right? Is the line vertical or horizontal? He objected that the task was absurd, that he was just guessing. He guessed correctly far more often than chance allows.

This is blindsight. Vision without the experience of seeing. It has been studied for fifty years and the phenomenon itself is not in doubt, though the mechanism has been argued over throughout. A review by Alan Cowey covers the long history of the debate [44].

The obvious explanation is that some visual information reaches the cortex by routes that bypass V1. There are such routes, through the superior colliculus and the pulvinar. For decades the collicular route was the standard account. Then a 2010 study did the decisive experiment, reversibly inactivating the lateral geniculate nucleus in monkeys with V1 lesions and finding that blindsight disappeared [45]. The residual vision depended on the thalamic relay, not only on the midbrain.

Something related happens with emotional stimuli. Faces expressing fear presented so that they are not consciously seen still drive activity in the amygdala, apparently through a subcortical route that does not require visual cortex at all [46]. A review collects the wider evidence for non-conscious perception of emotional signals [47]. This is one reason the amygdala can react to a threat before you have consciously identified it: it is not waiting for the cortex to finish.

There is even a dissenting position on whether V1 is required for conscious vision at all, arguing from unusual cases that it is not [48]. That view is a minority one, and it is included here because pretending otherwise would be tidier than the evidence.

What Breaks, and What It Looks Like

Everything described so far was learned from damage. It is worth setting out what that damage actually produces, because the pattern of deficits is itself an argument about how the cortex is organised.

The most common consequence of occipital damage is a field defect. Because each hemisphere handles the opposite half of the visual field, a stroke in one occipital lobe produces homonymous hemianopia: the same half of the field is missing in both eyes. Smaller lesions produce smaller holes, called scotomas, and because V1 is retinotopically mapped, the shape of the hole follows the shape of the damage. This is the direct clinical payoff of retinotopy, and it is why Inouye and Holmes could reconstruct a cortical map from wounds a century ago.

Damage restricted to colour-selective regions produces cerebral achromatopsia. The eye is working and the retina's colour receptors are intact, but the world arrives drained of colour. It is not colour blindness in the inherited sense, which is a retinal condition. It is colour vision removed at the cortical level, sometimes in only part of the visual field.

Damage to motion-selective cortex produces akinetopsia, described earlier through patient LM [15].

Damage further along the ventral stream produces the agnosias, and these are the deficits that sound least believable until you see them. Visual form agnosia, DF's condition, removes the ability to derive shape from vision while leaving vision itself measurably intact [2]. Prosopagnosia removes the ability to recognise faces specifically, and can be severe enough that people fail to recognise close family, or themselves in a mirror, while recognising the same people instantly by voice or by gait. That such a narrow loss is possible at all is part of the evidence for the face-selective region in the fusiform gyrus [24].

Strangest of all is Anton-Babinski syndrome, in which a person is cortically blind and denies it. They will describe a room they cannot see, confabulating details, and explain away their collisions with furniture. The denial appears sincere rather than evasive. Whatever process normally reports the state of your own vision back to you has been damaged along with vision itself.

Then optic ataxia, in the parietal lobe, which is the mirror of visual form agnosia and which anchored the second half of the argument above [3].

Read as a set, these deficits say something the anatomy alone does not. Vision is not one faculty that degrades gracefully. It is an assembly of separable operations, and damage removes them one at a time, cleanly enough that each one can be missing while the rest carry on.

What the Visual Cortex Does When There Is Nothing to See

The visual cortex is not fixed hardware. Two lines of evidence make that clear, and they run in opposite directions.

The first is the critical period. In 1963, Torsten Wiesel and David Hubel sewed shut one eyelid in kittens for a period during early development, then reopened it. The eye was physically fine. The cortex was not: cells that should have responded to that eye no longer did, and the territory had been taken over by the other eye [49]. By 1970 they had mapped the window during which this vulnerability exists [50]. Deprivation outside that window did far less.

This is the biological basis of amblyopia, sometimes called lazy eye, and it is why paediatric ophthalmologists treat it urgently in young children. It is also where the textbook story tends to overreach. A 2020 review argues that the classical picture of a window that slams shut is too absolute, and that adult amblyopia is considerably more treatable than the older account implied [51]. The window narrows. It does not seal.

The second line runs the other way. What happens to visual cortex in people who have never had vision at all?

It does not go quiet. In 1996 imaging of blind participants reading Braille found activation in primary visual cortex during tactile reading [52]. Later work went further, finding that activity in early visual cortex in blind individuals correlated with verbal memory performance [53], and that V1 activation in congenitally blind people was associated with episodic retrieval [54]. A review of reorganisation following sensory loss collects the broader picture [55].

Tissue that in a sighted brain handles edges and orientations is, in a brain that never received visual input, doing verbal and memory work. Whatever V1 is, it is not committed to vision by its wiring alone.

A Note on What the Evidence Can Carry

One caution before the end, because it applies to a great deal of what has been described here.

Much of the human evidence comes from functional MRI, and fMRI does not measure neurons firing. It measures a haemodynamic signal, a change in blood oxygenation that follows neural activity with a delay of seconds. A careful 2001 study recorded neural activity and the imaging signal simultaneously and worked out what the relationship actually is [56]. The signal is real and informative. It is also indirect, coarse in time, and not a picture of cells firing.

The technique has grown considerably more powerful. By 2005 researchers could decode which of several orientations a person was looking at from the pattern of activity across early visual cortex [57]. And theoretical work on what the ventral stream actually preserves has produced striking demonstrations, including images that are physically different but indistinguishable to peripheral vision because they preserve the same summary statistics [58].

The other caution is about context. Object recognition in the ventral stream is not a bottom-up pipeline that identifies things in isolation. Expectations and scene context shape recognition from early on [59]. Anyone who has failed to recognise a colleague in an unexpected place has felt the effect directly.

What It Adds Up To

The visual cortex takes an image the eye has already partly digested, throws away the picture, and rebuilds the world as a description. That description forks. One branch runs into the temporal lobe and constructs the visual world you experience and can talk about. The other runs into the parietal lobe and computes the transformations that put your hand in the right place.

The evidence that these can come apart is genuinely strong and rests on a double dissociation between two kinds of patient. The evidence that they are separate, independent systems is weaker than the diagrams suggest, and the researchers who proposed the model have said so themselves.

What survives is the more interesting claim anyway. Somewhere in the back of your head, a process is measuring the world accurately enough to guide your hand, continuously, and never showing you its work. Reaching for a cup, you are not consulting the picture you see. You are relying on a second measurement you have no access to, made by tissue that can be destroyed while your experience of the cup goes on exactly as before.

The seeing you know about is not the only seeing you do.