Julie Carpenter doesn’t blame human users at all for anthropomorphizing chatbots. The longtime human-AI interaction expert and author of The Naked Android sees generative AI tools, like ChatGPT, as intentionally designed to elicit these reactions from users through highly personalized memories and interactions that replicate social experiences.

These one-sided connections may become even more complicated, and potentially disastrous, when AI-generated erotica is added as an option. Carpenter is apprehensive about OpenAI’s upcoming plans to allow adults to sext with ChatGPT. “I’m for people exploring their sexuality, but I want it in a way that’s safe for them as well as fun,” she says. “People have to be very aware that there’s a surveillance aspect to the data.”

OpenAI first hinted at allowing adult users to generate erotica using ChatGPT two years ago in an official document describing how its models are designed to behave. While the company still seems to be planning its eventual release, the timeline is murky. In recent months, OpenAI’s advisory council of outside experts expressed concerns about the risks of this imminent release, including the possibility of a “sexy suicide coach,” according to a report from The Wall Street Journal. OpenAI declined to comment when reached by WIRED.

People talking dirty to machines, like ChatGPT, is nothing new. “They want to monetize something they see that people are going to try and do anyway,” says Kate Devlin, a professor of AI and society at King’s College London, whose research involves digital sex.

What’s different here is OpenAI’s potential embrace of smut on a mainstream platform. Erotic chatbots are a revenue-driving business, but they are often relegated to niche sites with unremarkable names or Elon Musk’s Grok, known for nonconsensual deepfake images and going into a “MechaHitler” mode.

When the adult mode is released, it’ll be interesting to see what exactly ChatGPT will remember about the sexual proclivities of users. Over the past couple of years, OpenAI has continued to improve the chatbot’s memory feature, which logs user preferences and responds with personalized outputs. Are you a vegan? ChatGPT won’t recommend a steakhouse when asked for restaurant ideas. Want to go on a hike? It knows you’re in San Francisco from past chats, and will suggest reasonably close trails, even if that location context is not included in the prompt.

But, what happens when this type of data logging and hyper-specific personalization meets the horny sexting habits of millions of adult ChatGPT users? Erotic interactions with ChatGPT will further heighten the stakes for user privacy.

Rather than remembering your dietary preferences or what type of movies you enjoy the most, ChatGPT could recall how much you fantasized about furry porn, threesomes at the beach, and piss play, as well as tailor its daily responses with these sexual details in mind. It remains unclear how OpenAI plans to handle these mature conversations.

People who choose to interact with the smut version of ChatGPT at launch may find solace in the “temporary chats” feature. When enabled, these conversations are not shown in a user’s log history or used to improve OpenAI’s models.

But these chats are not as ephemeral as they may feel for users. “For safety purposes, we may still keep a copy for up to 30 days,” reads OpenAI’s website. There’s also a disclaimer included on the site page claiming that “data retention for certain services may be affected by recent legal developments.”