A Footnote on the Imagery Debate: Pylyshyn responds to Kosslyn; Kosslyn Replies


The following occurs in "What Shape Are A German Shepard's Ears", A Talk With Stephen M. Kosslyn on Edge [7.15.02].

"lately there are signs that interest in mental imagery might be picking up again. This might be a result of another round in my old debate with Zenon Pylyshyn. He's a good friend of Jerry Fodor, but unlike Fodor, Pylyshyn has maintained forever that the experience of mental images is like heat thrown off by a light bulb when you're reading: It's epiphenomenal, it plays no functional role in the process. Pylyshyn believes that mental images are just language-like representations and that it's an illusion that there's something different about them. He published his first paper in 1973. Jim Pomerantz and I replied to it in 1977 and the debate has just been rolling along ever since. Pylyshyn has great distain for neuroscience, to put it mildly. He thinks it's useless, and has no bearing at all on the mind.

"I really don't know what brings him to this conclusion. I suspect it's because he is one of the few (less than 2 percent of the population) people who does not experience imagery. He apparently doesn't even "get" jokes that depend on images. He also probably rejects the very idea of imagery on the basis of his intuitions about computation, based on Von Neumann architecture. He's clearly aware that computers don't need pictorial depictive representations. His intuitions about the mind may be similar. But this is all speculation."

Apart from speculating that my views derive from the fact that I do not have mental images (I have vivid imagery and, being an academic who lives his life in his head, I use them all the time) and even that I fail to get the jokes in images (god knows where these irrelevant ideas come from but they deserve to be in gossip magazines rather than in a public document)! Kosslyn repeats (again and again) the old saw that I believe that images are epiphenomenal and play no role in thought. This gives away an assumption that Kosslyn shares with many people who support the picture-theory of mental imagery: the assumption that the subjective impression we have in mental imagery in itself constitutes a theory. If you claim for some X that X is "epiphenomenal" then you had better be prepared to explain exactly what you mean by X. A vague allusion to a subjective experience cannot be either epiphenomenal or not. Nobody could claim that images are epiphenomenal until they tell us what they think images are, beyond being a subjective experience. Kosslyn has a view about what the experience of having an image comes to: To have an image is to have in your head something that /looks like/ what you are imaging. He calls this relation of "looking like" /depiction/. Since it can't literally look like the dynamic 3D world the best we can hope for is that it is a picture of the world. Now this sort of object is not epiphenomenal, it is just nonexistent because the theory behind it is false.

I have in many writings shown that this assumption is not just a metaphor or a comfortable way of speaking, but it is an essential assumption upon which everything else Kosslyn says about imagery depends — including what one expects to find in the brain to support that view. If you believe in this idea strongly enough there is almost no limit to the sorts of neuroscience findings that you take to be support for this view — including the mere increased metabolic activity in the visual areas of the brain that may accompany imagining. As I have argued at length, even if one found actual pictures laid out in visual cortex this would not support the picture theory, That's because a layout of activity in visual cortex is the wrong kind of thing -- the wrong kind of pattern activity to underwrite mental imagery. Unlike mental images, such patterns (which, by the way, have not been observed beyond a general increase in activity in V1) would have to be two-dimensional, retinotopic (i.e., only a few degrees of visual angle in width), constantly changing, and so on — just as are the patterns put there by vision.

The rush to find neuroscience evidence to support the picture theory is so strong that Kosslyn even argues that low-level psychophysical properties, such as the oblique effect (whereby the resolution threshold for bars is better when the bars are horizontal or vertical than when they are at an oblique angle) occur with mental imagery and are easily explained by the fact that there are more cells sensitive to vertical and horizontal orientations than oblique orientations in visual cortex — an account that may well explain the oblique effect the case of vision. But it could not possibly explain the effect in imagery (assuming that there really is such an effect — it is not easy to do psychophysical experiments with images in which findings are uncontaminated by what the observers know would happen if they actually observed the world). The patterns, according to the picture theory, are projected onto the visual cortex. But the orientation-sensitive cells in the visual cortex are sensitive to the orientation of lines projected on the retina, not the orientation of activation patterns on the surface of the cortex. They are sensitive to retinal orientation because of the way photoreceptive retinal cells are connected, forming a template that is more sensitive to one orientation that another. Unless mental images are projected onto the retina, the numerical distribution of different orientation-sensitive cells in the visual cortex cannot explain any psychophysical effects found with mental images. But so desperate are those who seek a neurophysiological explanation of the bankrupt picture theory of mental imagery that they take almost any neuroscience finding as support. Contrary to Kosslyn's claim (quoted above) that I have "great disdain for neuroscience", what I have disdain for — not only in neuroscience but anywhere else in research — is poor reasoning and irrationally held beliefs. If Kosslyn does not have an adequate argument for his views he should at least keep from speculating on my motives and refrain from advertising his ideas about what I like or don't like in public documents.

For a short description of my views the reader may consult:

Pylyshyn, Z. W. (2003). Return of the Mental Image: Are there really pictures in the brain? Trends in Cognitive Sciences, 7(3), 113-118.

For a longer one and an open peer commentary:
Pylyshyn, Z. W. (2002). Mental Imagery: In search of a theory. Behavioral and Brain Sciences, 25(2), 157-237.

And for a work that puts this debate in perspective along with an analysis of vision:

Pylyshyn, Z. W. (2003). Seeing and visualizing: It's not what you think. Cambridge, MA: MIT Press/Bradford Books.

Pylyshyn, Z. W. (forthcoming, 2007). Things and Places: How the mind connects with the world (Jean Nicod Lectures Series). Cambridge, MA: MIT Press.

ZENON PYLYSHYN is Board of Governors Professor of Cognitive Science, Center for Cognitive Science and Department of Psychology, Rutgers University. His latest book is Seeing and Visualizing: It's not what you think.

Zenon Pylyshyn's Edge Bio Page


I apologize if my comments offended Zenon Pylyshyn. I was asked questions in the course of the interview, and responded honestly and directly on the basis of what I knew (or thought I knew, based in part on what I recalled Zenon's telling me, some 25 years ago — but I'll be the first to acknowledge that memories blur and morph with time, and I could have misremembered his comments).

The "debate" has now become far too personal, which is not in the interest of advancing science. My colleagues and I have replied to Zenon's scientific claims at length in our recent book, Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006), The Case for Mental Imagery. New York: Oxford University Press; we invite the interested reader to decide whether what Zenon says in his comment here is an accurate reflection of the empirical findings (e.g., concerning the activation of area V1), of our theory (e.g., of depictive representation), or indeed of views that he himself has previously expressed in print. Suffice it to say that 35+ years of hard work, which have led to increasingly detailed computer simulation models and scores of papers reporting empirical results, do not rely on "the assumption that the subjective impression we have in mental imagery in itself constitutes a theory" nor hinge on a "vague allusion to a subjective experience".

Stephen Kosslyn's Edge Bio Page

Return to Stephen M. Kosslyn's "What Shape Are A German Shepard's Ears"

John Brockman, Editor and Publisher
Russell Weinberger, Associate Publisher

contact: [email protected]
Copyright © 2007 By
Edge Foundation, Inc
All Rights Reserved.