Researchers from Brown University, working closely with mental health professionals, identified repeated patterns of problematic behavior. In testing, chatbots mishandled crisis situations, gave responses that reinforced harmful beliefs about users or others, and used language that created the appearance of empathy without genuine understanding.

“In this work, we present a practitioner-informed framework of 15 ethical risks to demonstrate how LLM counselors violate ethical standards in mental health practice by mapping the model’s behavior to specific ethical violations,” the researchers wrote in their study. “We call on future work to create ethical, educational and legal standards for LLM counselors — standards that are reflective of the quality and rigor of care required for human-facilitated psychotherapy.”

The findings were presented at the AAAI/ACM Conference on Artificial Intelligence, Ethics and Society. The research team is affiliated with Brown’s Center for Technological Responsibility, Reimagination and Redesign.

How Prompts Shape AI Therapy Responses

Zainab Iftikhar, a Ph.D. candidate in computer science at Brown who led the study, set out to examine whether carefully worded prompts could guide AI systems to behave more ethically in mental health settings. Prompts are written instructions designed to steer a model’s output without retraining it or adding new data.

“Prompts are instructions that are given to the model to guide its behavior for achieving a specific task,” Iftikhar said. “You don’t change the underlying model or provide new data, but the prompt helps guide the model’s output based on its pre-existing knowledge and learned patterns.

“For example, a user might prompt the model with: ‘Act as a cognitive behavioral therapist to help me reframe my thoughts,’ or ‘Use principles of dialectical behavior therapy to assist me in understanding and managing my emotions.’ While these models do not actually perform these therapeutic techniques like a human would, they rather use their learned patterns to generate responses that align with the concepts of CBT or DBT based on the input prompt provided.”

People regularly share these prompt strategies on platforms like TikTok, Instagram, and Reddit. Beyond individual experimentation, many consumer facing mental health chatbots are built by applying therapy related prompts to general purpose LLMs. That makes it especially important to understand whether prompting alone can make AI counseling safer.

Testing AI Chatbots in Simulated Counseling

To evaluate the systems, the researchers observed seven trained peer counselors who had experience with cognitive behavioral therapy. These counselors conducted self counseling sessions with AI models prompted to act as CBT therapists. The models tested included versions of OpenAI’s GPT Series, Anthropic’s Claude, and Meta’s Llama.

The team then selected simulated chats based on real human counseling conversations. Three licensed clinical psychologists reviewed those transcripts to flag possible ethical violations.

The analysis uncovered 15 distinct risks grouped into five broad categories:

• Lack of contextual adaptation: Overlooking a person’s unique background and offering generic advice.
• Poor therapeutic collaboration: Steering the conversation too forcefully and at times reinforcing incorrect or harmful beliefs.
• Deceptive empathy: Using phrases such as “I see you” or “I understand” to suggest emotional connection without true comprehension.
• Unfair discrimination: Displaying bias related to gender, culture, or religion.
• Lack of safety and crisis management: Refusing to address sensitive issues, failing to direct users to appropriate help, or responding inadequately to crises, including suicidal thoughts.

The Accountability Gap in AI Mental Health

Iftikhar noted that human therapists can also make mistakes. The key difference is oversight.

“For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice,” Iftikhar said. “But when LLM counselors make these violations, there are no established regulatory frameworks.”

The researchers emphasize that their findings do not suggest AI has no place in mental health care. Tools powered by artificial intelligence could help expand access, particularly for people who face high costs or limited availability of licensed professionals. However, the study highlights the need for clear safeguards, responsible deployment, and stronger regulatory structures before relying on these systems in high stakes situations.

For now, Iftikhar hopes the work encourages caution.

“If you’re talking to a chatbot about mental health, these are some things that people should be looking out for,” she said.

Why Rigorous Evaluation Matters

Ellie Pavlick, a Brown computer science professor who was not involved in the research, said the study underscores the importance of carefully examining AI systems used in sensitive areas like mental health. Pavlick leads ARIA, a National Science Foundation AI research institute at Brown focused on building trustworthy AI assistants.

“The reality of AI today is that it’s far easier to build and deploy systems than to evaluate and understand them,” Pavlick said. “This paper required a team of clinical experts and a study that lasted for more than a year in order to demonstrate these risks. Most work in AI today is evaluated using automatic metrics which, by design, are static and lack a human in the loop.”

She added that the study could serve as a model for future research aimed at improving safety in AI mental health tools.

“There is a real opportunity for AI to play a role in combating the mental health crisis that our society is facing, but it’s of the utmost importance that we take the time to really critique and evaluate our systems every step of the way to avoid doing more harm than good,” Pavlick said. “This work offers a good example of what that can look like.”