No-one really knows what we should be worried about. The smartest people around seem to generally think the answer is machine intelligence. They may be right, but other types of inhuman intelligence have an established track record well worth worrying about. Right now, I'm worried that so few smart people, whatever they are worried about, are doing anything to fight back against it.
In a pre-verbal tribe, most of the monkeys don't need to decide where they are going. They don't need to make their own decisions—because if they go off on their own, either the leader will kill them for rebelling, or a jaguar will kill them once they are separated from their pack. In such a situation, only pack leaders need to activate the messy symbolic intuitions that tell them what's out there and what needs to be done. As Abraham Maslow might say, only someone who sees themselves as a chief of their own, who has satisfied their need for esteem, is in a good position to self-actualize.
Maslow suggests that at one level of description a human consists of five programs. One escapes immediate danger, one seeks comfort and physical security, one finds a social context in which to be embedded, one builds esteem within that context, and one directs big-picture intentionality. As a general rule, when a given program reports satisfaction the next program is turned on and starts competing in its more subtle game.
Some of those programs allocate attention to things that can be understood fairly rigorously, like a cart, a plow, or a sword. Other programs allocate attention to more complicated things, such as the long-term alliances and reproductive opportunities within a tribe. The former programs might involve situational awareness and detailed planning, while the latter programs might operate via subtle and tacit pattern detection and automatic obedience to crude heuristics. If so, the latter programs might be fairly easily hacked. They might also offer a less fertile ground for the emergence of a Universal Turing Machine. Mechanical metaphors of solidity and shape might constitute a good substrate for digital, and thus potentially abstract cognition, while social metaphors for continuous and vague properties like weirdness, gravitas and sexiness might constitute a poor foundation for universal cognition. These programs seem to have been disfavored by history's great scientific innovators, who tend to make statements like "I do not know what I may appear to the world, but to myself I seem to have been only like a boy playing on the sea-shore, and diverting myself in now and then finding a smoother pebble..." or "What do you care what other people think", which sound like endorsements of physical over social cognition.
These considerations lead me to worry about the consequences of assisting a population in satisfying its physiological needs but not its love, belonging and esteem needs. Such a population would be at risk for underdevelopment of the programs for satisfying physiological and safety needs. That would be harmful if these functions are the basis for precise thought in general. It might be argued that the correct solution to this problem is to hurry people through their love, belonging and esteem needs, but that approach incurs further hazards.
Dacher Keltner and other psychologists recently showed that higher socioeconomic class—essentially higher satisfaction of esteem needs—leads to increased unethical behavior, while other research suggests that external efforts to boost self-esteem tend to produce anti-social narcissism. Research of this type fits in well with Jonathan Haidt's theories on the authoritarian dimension of moral cognition. It may not be a coincidence that one of the world's most popular religions is literally named "submission". In general, authoritarian impulses have always advised rulers to keep their subjects' bellies full… and pressed firmly against the ground. Steven Pinker has compellingly established the success of modern society in bringing us unprecedented peace. People, who don't fear for their safety, but who despair of ever achieving love or belonging, are the most submissive.
When we say that someone is 'smart', we largely mean that they learn quickly. Under circumstances where most students reach a seventh grade reading level, the 'gifted' students absorb the full content of their schooling. According to New York City's three-time teacher of the year John Taylor Gatto, that curriculum consists primarily of six lessons, but it seems to me that these lessons can be summarized in one word, submission.
Robert Altmeyer's research shows that for a population of authoritarian submissives, authoritarian dominators are a survival necessity. Since those who learn their school lessons are too submissive to guide their own lives, our society is forced to throw huge wads of money at the rare intelligent authoritarian dominants it can find, from derivative start-up founders to sociopathic Fortune 500 CEOs. However, with their attention placed on esteem, their concrete reasoning underdeveloped and their school curriculum poorly absorbed, such leaders aren't well positioned to create value. They can create some, by imperfectly imitating established models, but can't build the abstract models needed to innovate seriously. For such innovations, we depend on the few self-actualizers we still get; people who aren't starving for esteem. And that does not include the wealthy, the powerful, and the 'smart'; they learned their lessons well in school.