Most Singularitarians have positive visions about the future with artificial general intelligence, but James Hughes of the Institute for Ethics and Emerging Technologies sees a different future. In this talk from the 2007 Singularity Summit, he describes some ways that AGI will feel different and alien from us, and the negative effects that it could have on society.

While humans are the most familiar example of intelligence, organizations provide another form of intelligence that we can look to when we imagine how AGI will behave. Companies and governments know more than one person can know and replace their components the same way biological organisms do, but they don't act like human culture. Like organizations, AGI will both magnify human intentions and develop its own, that might or might not coincide with human interests.

Even if artificial intelligence doesn't surpass human intelligence, it can still pose problems for society. Combine a botnet that defends itself from security measures with the annoyance of dumb biological organisms like cockroaches and you have a lot of potential problems. Intelligent augmentation (IA) should be considered as an alternative to AI because of the benefits of having human brains as control systems.