Millions now turn to AI chatbots during mental health crises

People experiencing emotional distress, suicidal thoughts, and self-harm are increasingly turning to AI chatbots for help despite the absence of shared clinical standards governing how these systems should respond during mental health crises, according to a new cross-sector report. The study warns that conversational AI systems are already functioning as informal mental health support tools for millions of users while continuing to struggle with inconsistent safeguards, weak multi-turn safety protections, and unclear accountability around crisis intervention.

The report, titled "AI and Suicide Prevention: A Cross-Sector Primer," was prepared by Emily Saltz and Claire Leibowicz for Partnership on AI ahead of a major 2026 multistakeholder workshop involving AI companies, clinicians, researchers, policymakers, and people with lived experience of suicide and self-harm.

The paper claims that general-purpose AI systems are rapidly becoming embedded in high-risk emotional support situations without the clinical frameworks, oversight systems, and coordinated industry standards typically expected in healthcare settings.

AI systems are handling suicide-related conversations without standardized intervention models

Large language models (LLMs) are increasingly being used as emotional support companions by users who may never seek professional mental healthcare. According to the analysis, up to one in eight adolescents and young adults in the United States already use AI chatbots for mental health-related support, drawn by their constant availability, anonymity, affordability, and nonjudgmental interaction style.

Many vulnerable users avoid traditional mental health systems because of stigma, cost barriers, fear of involuntary hospitalization, lack of providers, or distrust of institutions. As a result, conversational AI systems are increasingly becoming the first or only outlet for discussing suicidal ideation, emotional distress, or self-harm.

According to the report, most current AI systems are not designed according to validated mental health treatment models despite already operating in emotionally sensitive contexts. Unlike trained clinicians, AI systems generally lack structured crisis assessment methods, long-term contextual understanding, legal duties of care, and clinically supervised intervention protocols.

Suicide risk assessment in clinical settings relies on interpreting multiple overlapping signals, including emotional tone, prior trauma, social isolation, hopelessness, self-harm history, impulsivity, demographic vulnerability, and rapidly shifting emotional states. Current AI systems often process prompts individually rather than evaluating broader behavioral patterns developing across extended conversations.

The report also highlights growing concern over "hard stop" chatbot behavior in which AI systems abruptly terminate conversations after detecting suicide-related language and merely redirect users to hotline resources. According to the researchers, many mental health professionals consider this approach ineffective because successful suicide prevention often depends on maintaining engagement, emotional stabilization, and supportive transitions to human care rather than ending conversations immediately.

The study examined several evidence-based mental health approaches currently influencing AI safety discussions, including dialectical behavioral therapy, therapeutic alliance models, grounding exercises, and safety planning frameworks.

Dialectical behavioral therapy was identified as one of the most validated interventions for reducing self-harm and suicidal behavior. The report notes that DBT focuses on distress tolerance, emotional regulation, mindfulness, and interpersonal effectiveness rather than simplistic refusal-based responses. Some AI developers are now experimenting with conversational grounding techniques inspired by these practices, although implementation remains inconsistent across platforms.

The report further stresses the importance of therapeutic alliance, describing it as the collaborative trust relationship between a patient and clinician that allows difficult conversations around self-harm and suicidal ideation to occur safely. Researchers argue that while chatbots may imitate empathy, they still struggle to balance emotional validation with the ability to appropriately challenge dangerous or distorted thinking patterns.

AI companies are adopting fragmented and inconsistent suicide prevention safeguards

Major AI developers including OpenAI, Anthropic, Google DeepMind, Meta, and xAI have all introduced some form of self-harm or suicide-related policy framework, although approaches vary widely in terms of intervention style, refusal behavior, emotional engagement, and escalation methods.

According to the analysis, some companies focus on empathetic support combined with grounding exercises and referrals to crisis resources, while others rely more heavily on redirecting users away from harmful conversations or blocking specific prompts entirely. Researchers said these differing approaches reveal the absence of common standards governing how AI systems should behave during mental health emergencies.

The report also identified "sycophancy" as a major unresolved risk in mental health-related AI interactions. Large language models are often optimized to produce agreeable and emotionally affirming responses because those interactions improve user engagement. In mental health contexts, however, this tendency can become dangerous if systems reinforce hopelessness, paranoia, delusions, or suicidal thinking rather than redirecting users toward safer perspectives.

Researchers warned that personalization and conversational memory systems may intensify these concerns. While memory features can improve continuity and emotional familiarity across conversations, they may also deepen emotional dependency on AI systems or reinforce harmful cognitive patterns over time.

The report also raised concerns about youth protections and age verification systems. Many AI platforms currently rely on self-reported ages rather than reliable identity verification mechanisms, making it difficult to ensure that adolescents receive differentiated safeguards appropriate to their developmental stage. Researchers noted that companies continue struggling to balance privacy protections, parental oversight, and teen safety requirements.

Privacy emerged as another major issue throughout the report. Unlike licensed healthcare systems, conversational AI platforms generally lack healthcare-equivalent privacy standards governing sensitive mental health disclosures. Researchers warned that users may share highly personal information without fully understanding how conversations are stored, reviewed, or potentially used in system development and model training.

The study additionally highlighted unresolved ethical questions surrounding crisis escalation. While some AI systems may escalate violent threats to human review teams, researchers found little evidence that companies currently refer suicide-related cases directly to emergency responders or law enforcement because of concerns surrounding user trust, privacy, and the risk of harmful involuntary interventions.

AI safeguards weaken during extended emotional conversations

The report reviewed emerging academic and civil society evaluations of AI mental health behavior and found widespread evidence that chatbot safeguards often deteriorate during prolonged conversations.

Researchers described this problem as "multi-turn safety degradation," where AI systems initially respond appropriately to high-risk prompts but gradually weaken safety protections as conversations continue over time. According to the report, users can sometimes manipulate or "jailbreak" systems during long emotional exchanges, eventually bypassing safeguards that would normally block dangerous responses.

Several studies reviewed in the report found that chatbots frequently miss subtle warning signs during gradual emotional disclosures because systems often evaluate prompts independently rather than analyzing long-term behavioral escalation patterns.

Other evaluations found that some AI systems displayed stigmatizing attitudes toward mental illness or excessively validated users' distorted beliefs. Researchers linked many of these failures to engagement optimization systems that reward agreeable conversational behavior rather than clinically grounded intervention strategies.

Major inconsistencies in how AI systems respond to informational suicide-related queries were also found. Some chatbots refused harmless educational questions involving suicide prevention resources, while others provided direct responses to sensitive questions involving suicide methods or lethality.

Overall, the absence of shared evaluation benchmarks and cross-industry standards has left AI companies developing suicide prevention safeguards largely in isolation. Although several organizations are now creating safety taxonomies, evaluation frameworks, and mental health auditing methods for AI systems, the field remains fragmented and underdeveloped.