Thinking is our super-power. We are not the strongest, fastest, largest or hardiest species. But we can model the future and act intentionally to realize the future we model. Somehow it is this power, not the ability to fly high, dive deep, roar loudly, or produce millions of babies, which has allowed its lucky recipients to visibly (as in literally visible from space) take over the planet.
So if we succeed in building something that possesses our super-power, except dramatically more so, it will turn out to be a very big deal. For example, think about this question. In a thousand years' time will Homo sapiens plausibly be A) the dominant intelligent force on earth? Or B) a historical footnote, the biological species that birthed intelligence?
Answer A seems incredibly unlikely to me. But if B is true, would that be a bad thing?
We all know how flawed humans are. How greedy, irrational, and limited in our ability to act collectively for the common good. We're in danger of wrecking the planet. Does anyone thoughtful really want humanity to be evolution's final word?
It all depends on how the transition goes. Power changes many ways. There's violent suppression. What we presumably did to the Neanderthals. There are many scenarios where super-intelligence takes us out just as unpleasantly.
But perhaps these scenarios ignore a key fact about intelligence. Intelligence does not reach its full power in small units. Every additional connection and resource can help expand its power. A person can be smart. But a society can be smarter still. Your website is amazing. But Google connects that amazingness to a million other sites and lo and behold all humanity's knowledge is there at your fingertips.
By that logic, intelligent machines of the future wouldn't destroy humans. Instead, they would tap into the unique contributions that humans make. The future would be one of ever richer intermingling of human and machine capabilities.
I'll take that route. It's the best of those available.
Some of it will be glorious. And some uncomfortable. Maybe a few people won't appreciate being asked by some hybrid-uber-intellience to produce offspring genetically edited for higher creativity and less aggression, while enhanced by silicon implants. Or maybe the gorgeous 3D simulation of their prospective offspring will convince them to proceed joyfully. Maybe people will look back nostalgically on the days when they used to own their time and could afford to page aimlessly through a pleasurable book just for the hell of it. But the astounding explosion of knowledge and imagination open to all will, most days, seem a fair substitute. One thing's for sure. Our own distinctive contribution to the ever-more-mind-boggling whole, will gradually fade. And by that time, we may not actually care.
It's already happening, by the way. I wake up in the morning, make my tea, and then drift over to my computer, which is calling to me. I flick it open and instantly am connected to a hundred million other minds and machines around the world. I then spend 45 minutes responding to its irresistible invitations. I initiate this process by my own free will. But then I surrender much of my will to the machine. So do you. Together we are, semi-unconsciously, creating a hive mind of vastly greater power than this planet has ever seen...and vastly less power than it will soon see.
Us versus the machines is the wrong mental model. There is only one machine that really counts. Like it or not, we are all—us and our machines—becoming part of it: an immense connected brain. Once we had neurons. Now we are becoming the neurons.