Why officials want to go after platforms, not users

Officials “want to introduce a new ban on so-called ‘nudifier’ systems that use AI to create or manipulate images that are sexually explicit or intimate and resemble an identifiable real person without that person’s consent,” the press release explained. However, “the ban would not apply to AI systems with effective safety measures preventing users from creating such images,” officials said.

As Bloomberg noted, the ban would radically shift the EU’s approach to regulating explicit deepfakes, moving beyond just prosecuting users to also punishing platforms. The Grok scandal “epitomized” why such a regulatory shift was needed, Bloomberg reported, noting that “this amendment is the first” EU policy “to specifically target AI platforms” that produce and allow sharing of “sexual material without the subject’s consent.”

While EU officials did not directly mention Grok in the press release, regulators had already been probing the AI system while pondering the implications of xAI’s controversy for other, less visible nudify apps. Submitting questions to the European Commission earlier this year, lawmakers warned:

Recent shocking reports of AI-powered nudity applications, such as Grok on X, but also other tools that are freely available online, highlight an increase in AI-driven tools that allow users to generate manipulated intimate images of individuals without their consent, facilitating gender-based cyberviolence and the creation of child sexual abuse material.

“These systems should be banned from the EU market,” lawmakers urged, particularly since “individual perpetrators”—who “can often be punished under national criminal law”—“are often hard to find.” A more proactive plan, lawmakers suggested, would be to “prevent widespread image-based sexual violence from the outset.”

With apparent backing from Parliament members, the amendment’s likely passage is sure to frustrate Musk, who is also facing legal challenges in the US seeking injunctions against Grok’s nudify outputs. In January, a mother of one of Musk’s children, Ashley St. Clair, became one of the first victims to file a lawsuit. And more recently, three young girls in Tennessee filed a proposed class action representing all children harmed by Grok’s alleged CSAM outputs.

In the EU, similar public pressure is mounting for regulators to intervene, as xAI seems unwilling prevent Grok from undressing real people. A civil liberties committee member, Michael McNamara, said in the press release that he believes the proposal to ban nudify apps “is something that our citizens expect.”