A pervasive, human fear emerged in the 20th century, one that grows stronger with each new doomsday prediction: inevitably, as artificial intelligence advances, some unforeseen computer bug will cause computers to revolt and take over the world.
My concern is actually the opposite: that as artificial intelligence advances, it will not be buggy enough. Machines designed to think that are perfectly self-correcting, self-optimizing, and self-perfecting—until the square peg always ends perfectly in the square hole—will also be machines that fail to inculcate the random sparks of insight that come from the human tendency to be "buggy": to try to fit square pegs into round holes, or even more broadly speaking, to notice the accidental but powerful insights that can arise as a byproduct of solving a shape/whole problem.
Consider the power of noticing. The reason we can enjoy macaroni and cheese in a matter of seconds? While working at Raytheon, Percy Spencer walked in front of a machine, and happened to notice that his chocolate bar had melted. Why? The machine generated microwaves. Rather than attempt to optimize the magnetron to avoid future chocolate failure, Spencer had the flash of insight to realize that the melted chocolate was the harbinger of something bigger.
Consider the power of accidents. Rubber was doomed to specialized usage due to its failure to withstand extreme temperatures—until Charles Goodyear slipped up and dropped some rubber on a hot stove. Rather than clean the stove and take steps to ensure no future mistakes, Goodyear noticed something interesting about the resulting product. The result was vulcanized, weatherproof rubber.
Finally, consider the power of human "bugs"—our biases. For example, optimism makes us believe we can get to the moon, cure all diseases, and start a successful business in a location whose previous tenant closed "only" because it's in a terrifying location. The endowment effect causes us to overvalue what we have, what we ideate, and what we create—even when no one else alive agrees. But is abandoning all endeavors at the first sign of failure and pursuing one that seems more successful always optimal? The dogged scientists (think some mildly famous ones like Galileo and Darwin) who persisted in the face of more generally accepted explanations were being stubborn—being buggy—but the result was genius.
I am a behavioral scientist, a field that has spent decades identifying bugs and attempting to debias them. But at the same time we must acknowledge: in large part, it's the bugs that make us—and, in the end, any form of intelligence—human.