As I was
doing this, it suddenly occurred to me that this is an interesting model
of mental imagery. We could think of imagery as having four main components:
Its got a deep representation, which is an abstract representation
in long-term memory; its got a surface representation, which is
like a display in a cathode ray tube; its got generative processes
between the two , so the surface geometry is reconstructed in the surface
image on the basis of the deep representation; and, finally, it's got
interpretative processes that run off the surface image, interpreting
the patterns as representing objects, parts or characteristics.
This metaphor was neat, and led me to conduct a lot of fruitful research.
But it had the drawback that no matter how hard you hit somebody in
the head, youre not going to hear the sound of breaking glassthere's
no screen in there. Even if there were, would just be be back to that
problem of the little man looking at the screen. This immediately led
me to start thinking about how to program a system where there are arrays
that function as a surface image and points that are positioned in space
depicting the pattern that represents an object. And then you have something
much more abstract that's operated on to produce that.
One of the real virtues of thinking by analogy to the computer is that
it focuses you on the idea of processing systems not just isolated
representations or processes, but sets of them working together. Nobody
had ever tried to work out in detail what a processing system that uses
images would look like. In fact, the few detailed models of imagery
that existed all focused on very specific, artificial tasks and tried
to model them using standard list-structures there were no images
in the models of imagery. We decided to take seriously the idea that
perhaps mental images aren't represented the same way as language; perhaps
they really are images. Steve Schwartz and I built a series of simulation
models that showed such an approach is not only possible, but allows
you to account for much data. We published our first paper on this in
77, and another in 78. I also wrote a book on it in 1980
called Image in the Mind, where I worked this out in much more detail
than anyone ever cared about. As far as I can tell it had almost no
impact. I remember asking one my professors at Stanford about it, and
he thought the book was too detailed, and that for somebody to start
working on the topic now theyd have to look at it, think about
it and get into it, and it was just too much trouble. Psychologists
generally don't like having to work with a really detailed theoretical
framework, and that was basically the end of it. I have a mild frontal
lobe disorder that leads me to perseverate, and thus I've continued
to work out the theory and do experiments anyway. My 1994 book on imagery
is a direct outgrowth of the earlier work, but now maps it into the
brain. The Europeans (especially the French) and Japanese seem interested,
if not the Americans.
That said, I should note that lately there are signs that interest in
mental imagery might be picking up again. This might be a result of
another round in my old debate with Zenon Pylyshyn. Hes a good
friend of Jerry Fodor, but unlike Fodor, Pylyshyn has maintained forever
that the experience of mental images is like heat thrown off by a light
bulb when youre reading: It's epiphenomenal, it plays no functional
role in the process. Pylyshyn believes that mental images are just language-like
representations and that its an illusion that theres something
different about them. He published his first paper in 1973. Jim Pomerantz
and I replied to it in 1977 and the debate has just been rolling along
ever since. Pylyshyn has great distain for neuroscience, to put it mildly.
He thinks it's useless, and has no bearing at all on the mind.
I really dont know what brings him to this conclusion. I suspect
its because he is one of the few (less than 2 percent of the population)
people who does not experience imagery. He apparently doesn't even "get"
jokes that depend on images. He also probably rejects the very idea
of imagery on the basis of of his intuitions about computation, based
on Von Neumann architecture. He's clearly aware that computers dont
need pictorial depictive representations. His intuitions about the mind
may be similar. But this is all speculation.
Pylyshyn is not only against theories that are rooted in neural mechanisms
(he thinks theories of the logical structure of language should be a
model for all other types of theories
really!), he's also against
neural network computational models. I've probably published eight to
ten papers using network models. At one point in my career I did work
on the nature of spatial relations. I had the idea that there are actually
two ways to represent relations among objects. One is what I call categorical,
where a category defines an equivalence class. Some examples of this
would be "left of," "right of," "above,"
"below," "inside," "outside," etc. If
you are sitting across from me, from your point of view, this fist is
to the right of this open palm, and that is true for all these different
positions [moving his hand about, always to the right of the vertical
axis created by his fist]. "Right of" defines a category,
and even though I move my hand around, all of these positions are treated
This is useful for doing something like recognizing a hand, since the
categorical spatial relations among my fingers, palm, digits, and joints
do not change. That's handy for recognizing objects because if you store
a literal picture in memory, an open-palm gesture might match well,
but if I make another gesture with my hand, say clenching it, this would
not match well so you want something more abstract.
Categorical spatial relations are useful computationally for that problem,
but theyre not useful at all for reaching or navigating. Just
knowing that a fist is to the left of this palm wont allow me
to touch it precisely; Ive got to know its exact location in space.
If Im walking around the room knowing the tables in front
of me, "in front of" is a categorical relation and thus is
true for an infinite number of positions relative to it. This is not
good enough for navigating. Thus, I posit a second type of spatial relation,
which I call coordinate: Relative to some origin, the metric distance
and direction is specified.