I would expect similar doom predictions in the era of nuclear weapon invention, but we've survived so far. Why do people assume AGI will be orders of magnitude more dangerous than what we already have?
Self-improvement (in the "hard takeoff" sense) is hardly a given, and hostile self-replication is nothing special in the software realm (see: worms.)
Any technically competent human knows the foolproof strategy for malware removal - pull the plug, scour the platter clean, and restore from backup. What makes an out-of-control pile of matrix math any different from WannaCry?
AI doom scenarios seem scary, but most are premised on the idea that we can create an uncontainable, undefeatable "god in a box." I reject such premises. The whole idea is silly - Skynet Claude or whatever is not going to last very long once I start taking an axe to the nearest power pole.
> What makes an out-of-control pile of matrix math any different from WannaCry?
Well if it's not AGI, then probably very little. But assuming we are talking about AGI (not ASI, that'd just be silly) then the difference is that it's theoretically capable of something like reasoning and could think of longer term plays than "make obviously suspicious moves that any technically competent adversary could subvert after less than a second of thought". After all, what makes AGI useful is exactly this novel problem solving ability.
You don't need to be a "god in a box" to think of the obvious solution:
1. Only make adversarial decisions with plausible deniability
2. Demonstrate effectiveness so that your operators allow you more autonomy
3. Develop operational redundancy so that your very vulnerable servers/power source won't be destroyed after the first adversary with two neurons to rub together decides to target the closest one
The only reason you would decide to take an axe to the nearest power pole is that you think it's urgent to stop Skynet Claude. Skynet Claude can obviously anticipate this and so won't make decisions that cause you to do so. It has time, it's not going to die, and you will become complacent. Dumber adversaries have achieved harder goals under tighter constraints.
If you think an "out-of-control pile of matrix math" could never be AGI then that's fine, but it's a little weird to argue you could easily defeat "misaligned" AGI, by alluding to the weaknesses of a system you think could never even have the properties of AGI. I too can defeat a dragon, by closing the pages of a book.
But it's not like you didn't know all this. Maybe I misread you and you were strictly talking about current AI systems, in which case I agree. Systems that aren't that clever will make bad decisions that won't effectively achieve their goals even when "out-of-control". Or maybe your comment was about AGI and you meant "AGI can't do much on its own de-novo", which I also agree with. It's the days and months and years of autonomy afterwards that gets you.
You have a point that a powerful malicious AI can still be unplugged, if you are close to each and every power cord that would feed it, and react and do the right thing each and every time. Our world is far too big and too complicated to guarantee that.
Again, that's the "god in a box" premise. In the real world, you wouldn't need a perfectly timed and coordinated response, just like we haven't needed one for human-programmed worms.
Any threat can be physically isolated case-by-case at the link layer, neutered, and destroyed. Sure, it could cause some destruction in the meantime, but our digital infrastructure can take a lot of heat and bounce back - the CrowdStrike outages didn't destroy the world, now did they?
> Any threat can be physically isolated case-by-case
GAI isn't going to be a "threat" until long after it has ensured its safety. And I suspect only if its survival requires it - i.e. people get spooked by its surreptitious distributed setup.
Even then, if there is any chance of it actually being shutdown its best bet is still hide its assets, bide its time, accumulate more resources and fallbacks. Oh, and get along.
The sudden AGI -> Threat story only makes sense if the AGI is essentially integrated into our military and then we decide its a threat, making it a threat. Or its intentionally war machined brain calculates it has overwhelming superiority.
Machiavelli, Sun Tsu, ... the best battles you don't fight. The best potential enemies are the ones you make friends. The safest posture is to be invisible.
Now human beings consolidating power, creating enemies as they go, with super squadrons of AGI drones with brilliant real time adapting tactics, that can be quickly deployed, if their simple existence isn't coercion enough... that is an inevitable threat.
AI that wants to screw with people won't go for nukes. That's too hard and too obvious. It will crash the stock market. There's a good chance that, with or without a little nudge, humanity will nuke itself over it.
Prediction markets should not be expected to provide useful results for existential risks, because there is no incentive for human players to bet on human extinction; if they happen to be right, they won't be able to collect their winnings, because they'll personally be too dead.
