As I was doing this, it suddenly occurred to me that this is an interesting model of mental imagery. We could think of imagery as having four main components: It’s got a deep representation, which is an abstract representation in long-term memory; it’s got a surface representation, which is like a display in a cathode ray tube; it’s got generative processes between the two , so the surface geometry is reconstructed in the surface image on the basis of the deep representation; and, finally, it's got interpretative processes that run off the surface image, interpreting the patterns as representing objects, parts or characteristics.

This metaphor was neat, and led me to conduct a lot of fruitful research. But it had the drawback that no matter how hard you hit somebody in the head, you’re not going to hear the sound of breaking glass—there's no screen in there. Even if there were, would just be be back to that problem of the little man looking at the screen. This immediately led me to start thinking about how to program a system where there are arrays that function as a surface image and points that are positioned in space depicting the pattern that represents an object. And then you have something much more abstract that's operated on to produce that.

One of the real virtues of thinking by analogy to the computer is that it focuses you on the idea of processing systems — not just isolated representations or processes, but sets of them working together. Nobody had ever tried to work out in detail what a processing system that uses images would look like. In fact, the few detailed models of imagery that existed all focused on very specific, artificial tasks and tried to model them using standard list-structures — there were no images in the models of imagery. We decided to take seriously the idea that perhaps mental images aren't represented the same way as language; perhaps they really are images. Steve Schwartz and I built a series of simulation models that showed such an approach is not only possible, but allows you to account for much data. We published our first paper on this in ’77, and another in ’78. I also wrote a book on it in 1980 called Image in the Mind, where I worked this out in much more detail than anyone ever cared about. As far as I can tell it had almost no impact. I remember asking one my professors at Stanford about it, and he thought the book was too detailed, and that for somebody to start working on the topic now they’d have to look at it, think about it and get into it, and it was just too much trouble. Psychologists generally don't like having to work with a really detailed theoretical framework, and that was basically the end of it. I have a mild frontal lobe disorder that leads me to perseverate, and thus I've continued to work out the theory and do experiments anyway. My 1994 book on imagery is a direct outgrowth of the earlier work, but now maps it into the brain. The Europeans (especially the French) and Japanese seem interested, if not the Americans.

That said, I should note that lately there are signs that interest in mental imagery might be picking up again. This might be a result of another round in my old debate with Zenon Pylyshyn. He’s a good friend of Jerry Fodor, but unlike Fodor, Pylyshyn has maintained forever that the experience of mental images is like heat thrown off by a light bulb when you’re reading: It's epiphenomenal, it plays no functional role in the process. Pylyshyn believes that mental images are just language-like representations and that it’s an illusion that there’s something different about them. He published his first paper in 1973. Jim Pomerantz and I replied to it in 1977 and the debate has just been rolling along ever since. Pylyshyn has great distain for neuroscience, to put it mildly. He thinks it's useless, and has no bearing at all on the mind.

I really don’t know what brings him to this conclusion. I suspect it’s because he is one of the few (less than 2 percent of the population) people who does not experience imagery. He apparently doesn't even "get" jokes that depend on images. He also probably rejects the very idea of imagery on the basis of of his intuitions about computation, based on Von Neumann architecture. He's clearly aware that computers don’t need pictorial depictive representations. His intuitions about the mind may be similar. But this is all speculation.

Pylyshyn is not only against theories that are rooted in neural mechanisms (he thinks theories of the logical structure of language should be a model for all other types of theories… really!), he's also against neural network computational models. I've probably published eight to ten papers using network models. At one point in my career I did work on the nature of spatial relations. I had the idea that there are actually two ways to represent relations among objects. One is what I call categorical, where a category defines an equivalence class. Some examples of this would be "left of," "right of," "above," "below," "inside," "outside," etc. If you are sitting across from me, from your point of view, this fist is to the right of this open palm, and that is true for all these different positions [moving his hand about, always to the right of the vertical axis created by his fist]. "Right of" defines a category, and even though I move my hand around, all of these positions are treated as equivalent.

This is useful for doing something like recognizing a hand, since the categorical spatial relations among my fingers, palm, digits, and joints do not change. That's handy for recognizing objects because if you store a literal picture in memory, an open-palm gesture might match well, but if I make another gesture with my hand, say clenching it, this would not match well — so you want something more abstract.

Categorical spatial relations are useful computationally for that problem, but they’re not useful at all for reaching or navigating. Just knowing that a fist is to the left of this palm won’t allow me to touch it precisely; I’ve got to know its exact location in space. If I’m walking around the room knowing the table’s in front of me, "in front of" is a categorical relation and thus is true for an infinite number of positions relative to it. This is not good enough for navigating. Thus, I posit a second type of spatial relation, which I call coordinate: Relative to some origin, the metric distance and direction is specified.