A computer is based on a Von Neumann architecture, where you have a strict separation between memory and the central processing unit. This means that there is a strict separation between operations and representations, which sit passively in memory. The central processing unit is essentially a switching device that uses instructions to dictate what it’s going to do, both in terms of how it interprets successive sets of instructions and what it does with the representations. The very idea of representation depends on how the CPU is set. The exact same pattern of bytes can represent a number, a letter, or part of a picture depending on how it’s being interpreted. Once an operation is performed, the results go back into memory, and serve as input for additional processes. The computer is useful as a way of thinking about all of this, but it’s not going to turn out to be a model of how the brain works; the brain doesn’t work like this at all.

The critical thing about the computer — in thinking about computation as a model for understanding the brain — is that it really lets us think about how advanced the mutual interaction of different levels of analysis is. It’s a wonderfulmystery. How can an idea arise from wet stuff? How can an idea influence what’s going on with the wet stuff? Here, the analogy really works with the computer. We really can think about the notion of representation in the computer, and how it dictates the sequence of physical events through the organization of instructions.

Even though I try to track developments in computational ideas, I'm not known as somebody who espouses the computational model of mind. Nor am I considered a neuroscientist. In fact, as far as I can see, I’m not known as a card-carrying member of any particular approach or subfield. I’ve always been on the fringe. When I was a graduate student I stumbled onto the basic phenomenon I've been studying for 30-odd years now. In my first year of graduate school at Stanford—this was 1970—the idea of semantic memory was really hot. Collins and Quillian had published a simulation model in 1969 in which they claimed that information is stored in long-term memory in the most efficient way possible. (This makes no sense for the brain, by the way, since storage space is apparently not an issue although it is in a computer.) They posited that memories are organized into hierarchies in which you store information in as general a representation as possible. For example, for animals, you've got a representation of animals, and then birds, mammals, reptiles, etc. And then under birds you have canaries, robins, etc. The notion was that you store properties as high up in the hierarchy as you can rather than redundantly duplicating them. For example, birds eat, but so do lizards and dogs, so we store this property higher, up with the concept of animals. You tag the exceptions in a lower level.

One way to test this theory was to look at response times. If you give somebody a statement like "A canary can sing," that information should be stored right in the same place and "canary" and "sing" should be bound together. But if you ask him, "Can a canary eat?" he should have to traverse the network to find a connection between the two (assuming that "eat" is stored up with "animal"). It should take a little longer, and it does! Unfortunately for the model, distance in a semantic net turned out not to be crucial. My first year project at Stanford showed that the response time was just due to how closely associated the terms were, not distance in a net.

One of the experiments was particularly interesting. In one item I asked people to verify the statement: "A flea can bite—true or false?" Two people in a row said false, and afterwards I asked them why. One said that he "looked for" a mouth, and couldn’t find one. The other said he "looked for" teeth and couldn’t "see" any. This idea of "looking for" and "seeing" didn’t fit in at all with Collins and Quillian's network-based computer model, so I started thinking about it. My idea was that maybe imagery has something to do with this. I telephoned everybody whom I’d tested already, and asked them if they had tended to visualize when they were answering the question. Roughly half said they did and about half said they didn’t. I simply plotted the data separately for the two groups. What was dictating the response time of the people who said they didn’t use imagery was how associated the properties were with the objects. For the people who used imagery, that had nothing to do it—it was how big the properties were.

I immediately designed an experiment where I pitted the two characteristics against each other. For example, I asked people, "True or false?: A mouse has a back," which is a trait that is big but not highly associated. I also asked whether it has whiskers, which is small and highly associated, or wings, which is not true. I found that if I instructed people to visualize, the critical thing was how big the properties were. The bigger they were the faster the responses were. If I asked them not to visualize, but to answer intuitively as fast as they could, the pattern reversed. In this case, the response speed depended on how associated the traits were, not how big they were.

The next question was how to think about these results. Fortuitously, at the same time I was doing these experiments I was taking a programming class. This was in the days when you used punch cards. You had to go to the computer center, submit your stack of cards, and stand around looking at a monitor, waiting for your job to come up and see whether it bombed, which you could tell by how long it was running. At the end they gave you a big printout. One of the exercises in the class was to program a set of little modules that generated geometric shapes, like triangles and squares and circles, and to adjust how big they were and where they were positioned. You had to do things like make a Christmas tree by recursively calling the same routine that generated a triangle, and plotting the triangle at different sizes in different positions, overlapping them to produce the design.