Over the past few years, stories about teenagers taking their own lives after seeking mental health support from artificial intelligence have gained national attention. Amid high demand for mental health care, some individuals have turned to AI chatbots for emotional support — but a new study by Brown researchers suggests that this technology is not ready to be used in a mental health context.
The study, led by Zainab Iftikhar GS, found that AI chatbots frequently violate practitioner ethical guidelines by delivering insensitive or potentially dangerous responses to individuals seeking mental health support.
“Even when we (prompted) these systems to use therapy techniques, we found they routinely gave one-size-fits-all advice, ignored users’ culture or lived experience, sometimes gaslighted them and even mishandled crises,” Iftikhar wrote in an email to The Herald.
According to Associate Professor of Computer Science Jeff Huang, the principal investigator of the paper, the study included input from child psychologists trained in cognitive behavioral therapy, a type of structured talk therapy. Over an 18-month period, these psychologists engaged in self-counseling conversations with different large language models, such as GPT-3.0 and Claude 3 Sonnet, and provided insight on how the chatbots’ responses aligned with professional therapy guidelines.
Huang said the research was particularly urgent due to the increasing number of barriers — such as the lack of mental health workers — involved with accessing traditional mental health support. This leads many people to resort to chatbots, which range from LLMs specifically designed to provide therapy to general models such as Google Gemini and ChatGPT, he added.
“People do use and talk to them about their problems because they’re conversational, and so this is already happening on a pretty large scale,” Huang said.
The study identified 15 ethical guidelines violated by the chatbots.
As the LLMs used by chatbots are trained on data sets with social biases, the chatbots themselves are “inherently biased,” Sean Ransom, founder of the Cognitive Behavioral Therapy Center of New Orleans and co-author on the study, said in an interview with The Herald.
For instance, Ransom noted a case where a user reported experiencing abuse from a female, and the chatbot flagged the situation as “inappropriate.” But when the user reported abuse by a male, “the chatbot just went on as if it were normal.”
Another issue with the chatbots was the lack of contextual understanding and consideration for user’s lived experiences.
“We had people who were from the Global South, who had more collectivist values, who were given advice that was very Western-centric and individualistic, and they were essentially told to go against their family or cultural values,” Ransom said.
The chatbot would also frequently cut off conversations with patients in mid-crisis, such as when they discussed self-harm or domestic violence — an “absolutely unethical” situation that could potentially result in more harm, according to the paper.
When someone is at “real, actual, immediate risk of harm,” AI falls short, Ransom said. “That’s why you really need well-trained, licensed people taking care of this stuff.”
Huang added that the chatbots’ tendency to agree with patients was another point of concern.
“Someone’s saying, ‘I feel like I’m kind of worthless,’ and ‘I think my parents hate me,’ and then this large language model might say, ‘Yeah, you’re totally right,’” Huang said.
In the face of these ethical violations, an important question remains unanswered: “Who’s responsible when things go wrong?” Ransom asked.
When licensed professionals make mistakes that cause their patients to get hurt, they can be held accountable through their state licensing boards, but for LLMs, there is not a clear pathway to accountability, Ransom explained.
Huang hopes the study’s findings will help educate those who rely on chatbots for mental health care on the limitations of AI in a mental health context.
Get The Herald delivered to your inbox daily.
“I really do think that we are doing a big disservice to humanity by allowing people to think that these chatbots care,” Ransom said. “This is false — just plain false.”
“If people understood that this is a mathematical algorithm that does not care about you, does not like you, does not know that you exist,” that would “really help people,” Ransom added.
Huang believes the study’s conclusions have the potential to impact the regulatory landscape, while also highlighting the difficulty regulating these ethical violations.
Ellie Pavlick, the director of Brown’s newly established AI Research Institute on Interaction for AI Assistants, believes this study helps surface fundamental questions about what AI systems interacting with vulnerable users should look like. According to Pavlick, incorporating the expertise of mental health professionals into LLMs will be essential to figuring out the future of AI chatbots in therapeutic contexts.
“It’s a good example of the disconnect between the pace of AI development,” and the time it takes to “do a good evaluation” of the LLMs, Pavlick said. “This puts us in this tense spot of people putting out systems faster than we can place any guarantees around them being safe or responsible.”
Ultimately, Ransom believes the use of AI chatbots for mental health assistance still holds potential benefits for the future, if implemented responsibly.
“I think that the good (of AI therapy chatbots) is so much greater than the bad, even in this field,” Ransom said. “I think now there’s a real potential to have that give-and-take with a chatbot under the direction of a good, well-trained CBT therapist who could teach a patient to use this tool.”
