ELIZA'S WORLD

What is the compelling urgency of the machine that it can so intrude itself into the very stuff out of which man builds his world?

— Joseph Weizenbaum

Somehow I managed to miss, until just a few days ago, the news that Joseph Weizenbaum had died. He died of cancer on March 5, in his native Germany, at the age of 85. Coincidentally, I was in Germany that same day, giving a talk at the CeBIT technology show, and — strange but true — one of the books I had taken along on the trip was Weizenbaum's Computer Power and Human Reason.

Born in 1923, Weizenbaum left Germany with his family in 1936, to escape the Nazis, and came to America. After earning a degree in mathematics and working on programming some of the earliest mainframes, he spent most of his career as a professor of computer science at MIT. He became — to his chagrin — something of a celebrity in the 1960s when he wrote the Eliza software program, an early attempt at using a computer to simulate a person. Eliza was designed to mimic the conversational style of a psychotherapist, and many people who used the program found the conversations so realistic that they were convinced that Eliza had a capacity for empathy.

The reaction to Eliza startled Weizenbaum, and after much soul-searching he became, as John Markoff wrote in his New York Times obituary, a "heretic" in the computer-science world, raising uncomfortable questions about man's growing dependence on computers. Computer Power and Human Reason, published in 1976, remains one of the best books ever written about computing and its human implications. It's dated in some its details, but its messages seem as relevant, and as troubling, as ever. Weizenbaum argued, essentially, that computers impose a mechanistic point of view on their users — on us — and that that perspective can all too easily crowd out other, possibly more human, perspectives.

The influence of computers is hard to resist and even harder to escape, wrote Weizenbaum:

The computer becomes an indispensable component of any structure once it is so thoroughly integrated with the structure, so enmeshed in various vital substructures, that it can no longer be factored out without fatally impairing the whole structure. That is virtually a tautology. The utility of this tautology is that it can reawaken us to the possibility that some human actions, e.g., the introduction of computers into some complex human activities, may constitute an irreversible commitment. . . . The computer was not a prerequisite to the survival of modern society in the post-war period and beyond; its enthusiastic, uncritical embrace by the most "progressive" elements of American government, business, and industry quickly made it a resource essential to society's survival in the form that the computer itself had been instrumental in shaping.

The machine's influence shapes not only society's structures but the more intimate structures of the self. Under the sway of the ubiquitous, "indispensable" computer, we begin to take on its characteristics, to see the world, and ourselves, in the computer's (and its programmers') terms. We become ever further removed from the "direct experience" of nature, from the signals sent by our senses, and ever more encased in the self-contained world delineated and mediated by technology. It is, cautioned Weizenbaum, a perilous transformation:

Science and technology are sustained by their translations into power and control. To the extent that computers and computation may be counted as part of science and technology, they feed at the same table. The extreme phenomenon of the compulsive programmer teaches us that computers have the power to sustain megalomaniac fantasies. But the power of the computer is merely an extreme version of a power that is inherent in all self-validating systems of thought. Perhaps we are beginning to understand that the abstract systems — the games computer people can generate in their infinite freedom from the constraints that delimit the dreams of workers in the real world — may fail catastrophically when their rules are applied in earnest. We must also learn that the same danger is inherent in other magical systems that are equally detached from authentic human experience, and particularly in those sciences that insist they can capture the whole man in their abstract skeletal frameworks.

His own invention, Eliza, revealed to Weizenbaum the ease with which we will embrace a fabricated world. He spent the rest of his life trying to warn us away from the seductions of Eliza and her many friends. The quest may have been quixotic, but there was something heroic about it too.

JARON LANIER
Computer Scientist and Musician; Columnist, Discover Magazine

We have lost a lion of Computer Science. Joseph Weizenbaum’s life is proof that someone can be an absolute alpha-geek and a compassionate, soulful person at the same time. He displayed innovative courage in recognizing the seductive dangers of computation.

History will remember Weizenbaum as the clearest thinker about the philosophy of computation. A metaphysical confrontation dominated his interactions with the non-human centered mainstream. There were endless arguments about whether people were special in ways that cybernetic artifacts could never be. The mainstream preferred to sprinkle the magic dust of specialness on the “instruments,” as Weizenbaum put it, instead of people.

But there was a less metaphysical side of Weizenbaum’s thinking that is urgently applicable to the most pressing problems we all face right now. He warned that if you believe in computers too much, you lose touch with reality. That’s the real danger of the magic dust so liberally sprinkled by the mainstream. We pass this fallacy from the lab out into the world. This is what apparently happened to Wall Street traders in fomenting a series of massive financial failures. Computers can be used rather too easily to improve the efficiency with which we lie to ourselves. This is the side of Weizenbaum that I wish was better known.

We wouldn’t let a student become a professional medical researcher without learning about double blind experiments, control groups, placebos, the replication of results, and so on. Why is computer science given a unique pass that allows us to be soft on ourselves? Every computer science student should be trained in Weizenbaumian skepticism, and should try to pass that precious discipline along to the users of our inventions.

Weizenbaum’s legacy includes an unofficial minority school in computer science that has remained human-centered. A few of the other members, in my opinion, are David Gelernter, Ted Nelson, Terry Winograd, Alan Kay, and Ben Schneiderman.

Everything about computers has become associated with youth. Turing’s abstractions have been woven into a theater in which we can enjoy fantasies of eternal youth. We are fascinated by wiz kids and the latest young billionaires in Silicon Valley. We fantasize that we will be uploaded when the singularity arrives in order to become immortal, and so on. But when we look away from the stage for a moment, we realize that we computer scientists are ultimately people. We die.