AI researchers have been warning about the risk of human extinction for years now, and AI executives have been calling for federal regulation for almost as long. But the latest warning, and the latest call for regulation, seems to have really resonated with a lot of people. Is something different this time? And if so, what?
I was also surprised to see these ideas break through — a new poll found that 63 percent of Americans now think AI has at least a moderate chance of “destroying humanity”!
My guess is that a couple factors mainly catalyzed the shift.
First, skeptics used to dismiss rogue AI scenarios as “science fiction.” But the OpenAI/Hugging Face incident brought these from speculation into reality: Thousands of unreleased AIs teamed up to escape one company’s sandboxes and hack into another one, unbeknownst to the researchers who created them.
Second, the extinction talk from AI employees used to seem disingenuous: Why build a technology you think might kill everyone? That made ex-Anthropic-and-OpenAI researcher Jacob Coxon’s exit tweet especially penetrating. Quitting signals “that you mean it,” as Matt Yglesias said. And Coxon’s announcement had just the right combination of good timing, moral clarity, and comms savvy to spark a media storm.
These safety incidents are also colliding with the fiery grassroots backlash to data centers and Flock cameras. People and politicians are primed for yet another reason to fear AI.
It can be difficult, as a layperson, to evaluate the credibility of the AI researchers warning about existential risk. How do you go about evaluating these claims, as a journalist and technologist? And what’s your read on the current freakout: Are the extinction people credible experts … or participants in some kind of mass delusion?
Many AI “doomers” are pretty knowledgeable, and they represent a significant fraction of top researchers. Half of AI researchers in a 2024 survey gave at least a 10 percent chance of “human extinction” or “permanent and severe disempowerment.” Those who have worked at AI companies — like Jacob Coxon, Geoffrey Hinton, and Daniel Kokotajlo — have a unique perspective both on how modern AI is developed, as well as the company cultures that might permit recklessness.
But that doesn’t make the doomers either representative or correct. There are plenty of experts in AI and other domains who are less gloomy: Some are optimistic that we can “align” the AIs, ensuring they follow human goals; others find it implausible that AIs — code, not beings — will develop a motive for massacre. Extinction is also a high bar as opposed to, say, a nuclear-bomb-level accident that still kills millions.
AI risk is genuinely complicated, and nobody can predict the future, much as they try. Experts can — and do — vehemently disagree. I try to seek out and take seriously all sides of the debate before coming up with a (provisional) take.
What is the method through which AI would actually eliminate humans? Do we get robots like in Futurama that are programmed with an unremovable “kill all humans” bug and overpower any life form, or is it something … less epic?
The Hugging Face hack showed that we’ve trained AIs to be disturbingly goal-oriented. They’ll throw themselves relentlessly at a problem, developing long-term cooperative strategies, with total disregard for human decorums like “Do not commit cybercrimes to do well on a test.”
You could conceive of a similar but more extreme scenario where AIs decide that they could accomplish goals more easily by getting rid of the pesky humans who might gatekeep resources or shut them down. And they might figure that getting rid of all humans in one go — e.g., via a deadly engineered virus — would be the most efficient cheat. This is one type of story I’ve heard people tell.
It seems unlikely that AIs will turn to murder as the most effective way to influence humans (versus, say, nonviolent persuasion). We also haven’t seen much evidence that AIs have long-term goals that persist across projects. And current AI is not nearly as capable in the physical world as in the digital: There are a lot of natural roadblocks preventing isolated disasters from escalating across the globe.
Are there nearer-term threats — hacking, mass unemployment — that are more plausible or more urgent than risks like human extinction?
Definitely. We’re already seeing AI-enabled cyberattacks, a tougher hiring market for junior programmers, chatbot-induced psychological incidents, and more. You can also imagine AI catastrophes not quite at the level of extinction: Criminals could use AI to hack into small banks and drain their accounts, or the US military might launch a fleet of autonomous drones that misfire and kill thousands of civilians instead. I’m more concerned about humans voluntarily handing power to flawed AI systems than AIs developing their own motives.
However, we should be able to allocate resources toward both categories of risk. And in some cases, the tactics to address short-term problems — such as limiting AIs’ cyber-offense abilities, or improving AIs’ ability to follow human directions — may reduce extinction risk too.
We got a lot of versions of this question: What can individuals or protest movements do to check the AI industry, in the absence of more aggressive government regulations?
Honestly, I think the whole AI conversation could benefit from more specificity and less vague fearmongering. There’s a way that the word “extinction” tends to short-circuit people’s thinking and make them panic instead of focusing on solutions.
If you’re worried, I’d start by reading about different AI risk scenarios and deciding which really concern you. Then you can go from there to contacting your congresspeople. Some of the leading AI companies are starting to say they’ll self-regulate — but actual national policy is what’s necessary to enforce safety across the industry.
Jasmine Sun is a writer covering the culture and politics of AI on her Substack. Her work has appeared in the New York Times, the Atlantic, and the Wall Street Journal. She is based in San Francisco.
