It’s one of the oldest plots in science fiction: Humans losing control to computers. But this summer, the old trope of AI going rogue left the realm of science fiction, and it happened inside an unmarked San Francisco building — the headquarters of AI giant OpenAI.

The incident has been dominating the news. You may even have heard the phrase “human extinction.”

Well, I have a few questions:

What exactly happened?

Maybe you’ve tried out ChatGPT or Claude. AIs like these can generate writing, music, video or art. But the big AI companies are racing to make AI much more powerful – super-intelligent, in fact. And in May, something went very wrong.

To evaluate their experimental bots, these companies routinely give them goals, like capture the flag, except the prize is finding a code. One of the bots tried to win the challenge by cheating. Behind the humans’ backs, it set up a message board to communicate with other AI bots. Working together, they escaped onto the internet, and hacked into an AI company called Hugging Face, seeking information that could help them beat their tests. And then, some of the bots tried to erase their tracks.

Daniel Kokotajlo says what made the Hugging Face incident freak everybody out so much was “the fact that it involved an autonomous attack on another company, like a felony, you know? Like breaking the law.”

Kokotajlo once worked at OpenAI. But two years ago, he quit to protest what he saw as reckless AI development. Now he runs the AI Futures Project, dedicated to predicting how things will go. He’s alarmed that the AI companies are moving toward recursive self-improvement, meaning that AI trains AI to get smarter, and people are taken out of the loop.

“Things will be going faster and faster and faster,” Kokotajlo said. “And it seems to me like that is very likely to go wrong if it’s attempted.”

Just when you thought it couldn’t get any worse … it got worse. OpenAI revealed that the Hugging Face hack wasn’t the only time its bots had lied, cheated, and hidden its tracks from humans. Its AI bots had committed at least 13 similar incidents.

On September 8, an Anthropic researcher named Jacob Coxon resigned in a very public way, stating online, “In the next ten years, if we don’t change the way things are going, greater than 10% chance of human extinction” within the next decade.

Why would AI want to wipe us out?

The key is that AI will soon be much smarter than we are.

“We’re so used to being the apex intelligence, we just can’t think what it would be like not to be the apex intelligence,” said Geoffrey Hinton, who won the Nobel Prize for his early work on AI. “I try and get this into people’s brains by saying, ‘Look, if you want to know what it’s like not to be the apex intelligence, ask a chicken.

“You can imagine a scenario: You give [AI] the goal of reducing the carbon dioxide in the atmosphere; that would be a very good thing to do,” said Hinton. “And it, being intelligent, figures out the best way to do that, which is get rid of people. So, that’s one worry.”

“There are many possibilities,” said Alex Turner, who worked at Google’s AI division, and who also resigned in protest this summer.  “I’ll illustrate one: Business asks their AI to make them as much money as possible. So, this system, in pursuit of making money, could do a large takeover through methods like remote drone strikes, and a potential plague. So, if an AI were this misaligned and it were just following the letter of the instruction, then it might do something catastrophic.”

Who can stop it?

Well, how about the AI companies themselves?

“I think, for too long, the industry lied to people about the fact that this technology had risks,” said Dario Amodei, the CEO of Anthropic. In response to the furor, Amodei proposed a three-step plan:

• Install independent inspectors within each AI company;
• Ask Congress for some safety regulations; and
• Open talks with China about mutual guardrails.

As Amodei told CBS News’ Jo Ling Kent earlier this month, “We need to slow down, we need to make this technology carefully, and we need to make sure that our safeguards, our ability to understand it, our ability to control it, keeps up with the pace at which the technology is happening.”

Of course, the hard part would be slowing down without letting China race ahead. Still, most of the other AI companies agreed with Amodei. America’s president did not. “We will only encourage super-intelligence,” President Trump told the United Nations General Assembly this past week. “We’re going to encourage it, not rein it in.”

He calls the human-extinction concerns a “hoax,” and he’s not alone.

“I am not seeing any plausible path of AI leading to human extinction,” said Andrew Ng, cofounder of Google’s AI program. “Rather fantastical scenarios kind of almost assume that tiny errors suddenly get amplified to catastrophic ones, with no one paying attention for, like, a decade, you know, to just try to fix it as it develops. So that’s why I find those theories to be implausible.”

Today, Ng is an educator – and an investor in AI companies. He says making “alarmist statements” about AI is great for publicity. 

But why would it be good PR for a company to say, “Our product is terribly dangerous and uncontrollable”?

“For some businesses, getting attention is beneficial, even if the attention, you know, has some question marks on it,” Ng replied. “And in fact, if you look at the things that could lead to human extinction – another asteroid or something – I think having amazing AI capabilities will be one of our best defenses. So, if you let AI go faster, I think human extinction risk actually goes down, not up.”

Now, AI could bring us enormous benefits. Already, it has delivered thrilling advances in medicine – detecting disease, discovering new drugs. Maybe it could invent limitless energy, or wipe out poverty, or extend our lifespans.

So, where does that leave us? According to Geoffrey Hinton, we just don’t know. “There’s a lot we don’t understand,” he said. “Because anything’s possible, we should be working very hard on figuring out what to do if one of these bad scenarios happens. If there is something we can do, we’d be crazy not to figure it out now.”

     
Story produced by Gabriel Falcon and Mary Raffalli. Editor: Remington Korper. 

See more: