“We’ve got a constant back and forth,” says Magnus. “We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them.” We’ll probably need to use AI to find ways to restrict the use of AI, he says.

I should add here that not all scientists agree on the level of risk. At a recent media briefing, some biologists at Imperial College London argued that AI tools just aren’t good enough to fully develop bioweapons, and that testing new pathogens requires difficult, time-consuming, human work. Some think the guardrails we have in place are sufficient.

And Wendy Barclay, a professor of infectious disease at Imperial, pointed out that, as things stand, the greatest risk of a pandemic isn’t from a bioweapon, but from pathogens that are already circulating. Take H5N1, the bird flu virus that has already killed millions of birds and spread widely through US dairy cattle; last month it was also detected in captive mink at a farm in Utah.

Sabra, on the other hand, likes to think five to 10 years ahead. Countries should be strengthening their health-care systems, preparing antidotes to known toxins, and stockpiling medicines, she says: “We need to be prepared.”

Kevin Esvelt, an MIT biologist who invented both technology to fast-track the propagation of a genetic feature through an entire population and ways to limit that technology, echoed these concerns in an X post on Wednesday, stating that a large language model had “disclosed a novel form of bioweapon that I hadn’t realized was possible.”

He added, “Please, for the love of God, children, the future of humanity, or whatever you consider holy, let’s err on the side of caution here.”

This article first appeared in The Checkup, MIT Technology Review’s weekly biotech newsletter. To receive it in your inbox every Thursday, and read articles like this first, sign up here.