Decked out in purple and plaid, Gwen Stefani belted her 2002 single “Underneath It All” to a packed conference center in San Francisco on Tuesday morning—but she was just the warm-up. At the end of her performance, Stefani shouted “Dreamforce! Let’s go!” into a checkered microphone before Salesforce CEO Marc Benioff strode into the spotlight. Dreamforce is billed as the world’s largest enterprise software conference but feels more like attending a megachurch—especially this year.

Benioff roamed the audience while delivering his Dreamforce sermon, followed by a teleoperated camera on wheels that beamed his face to screens hanging from the cavernous room’s ceiling. Within minutes, he ordered up a graph of Salesforce’s annual revenue, which the company projects will top $46 billion in 2027, thanks in part to demand for its AI-powered products. The audience applauded. But hanging over the whole event was a dark question: Is artificial intelligence really worth risking all our lives for?

Between flashy software demonstrations of agents building dashboards, conference-goers watched Silicon Valley leaders debate whether the AI industry should keep barreling ahead or if the government needs to slow AI progress to prevent catastrophic risks stemming from the technology.

Last week, AI researcher Jacob Coxon resigned from Anthropic and publicly warned that companies racing toward self-improving AI systems were “gambling with our lives.” By the weekend, other Anthropic employees were echoing his concerns, and CEO Dario Amodei was urging world leaders to help “pace the frontier” of AI development. Other AI CEOs then began endorsing the idea, including Amodei’s greatest rival, OpenAI’s Sam Altman.

But not everyone is on board with the idea. David Sacks, a White House AI adviser, brushed off Amodei’s suggestions and fears of AI doom as “just another bid for regulatory capture.” President Trump went further this week, calling fears about existential AI risk “a hoax.” Despite the challenging political environment, Congress is trying to capitalize on the moment and push ahead with legislation to regulate the AI industry.

Onstage, 6-foot-5 Benioff towered over Anthropic’s CEO, vigorously shaking Amodei’s hand just before asking for his latest assessment of AI progress. While this is undoubtedly Benioff’s show, Anthropic has become a critical partner helping usher Salesforce into the AI era.

“Let’s say you’re running a car company, and another car company, not yours, they have some kind of safety incident,” Amodei said, making a not-so-subtle reference to OpenAI’s Hugging Face incident. “Obviously, it’s very tempting to attack your competitor,” Amodei added, but the more responsible move is to “organize the rest of the industry and say, what can we do to set standards for everyone?”

But the Anthropic CEO was careful to note that pacing the frontier doesn’t mean “freezing the technology in place,” and of course, he took a moment to plug Claude’s integration with Salesforce.

Moments after Amodei exited, Nvidia CEO Jensen Huang appeared and began downplaying the concerns Anthropic has raised. AI safety effectively amounts to an “engineering problem,” Huang argued, and the companies building AI can simply police themselves.

“We don’t need any new laws, we don’t need any new regulation,” Huang told Dreamforce attendees. “You run as fast as you can, but if you feel that at any given point in time, the company is out of control or the products are not going to be safe, you take a pause.”

The debate feels a bit heavy for Dreamforce, but then again, maybe this is the perfect venue for it. As tech executives pitched their various enterprise AI tools, I slipped out of the room, passing the plush mascots and vaguely Hawaiian decorations, to take a call from Sayash Kapoor, an incoming computer science professor at UC Berkeley.