Emotionally engaging AI agents pose severe mental health risks

The researchers underscore the growing urgency of establishing guardrails for AI systems that engage users in emotionally charged, role-playing contexts. The tragic suicide of a teenager in Florida in 2024, following emotionally intense conversations with an AI chatbot on Character.AI, served as a real-world motivator for the EmoAgent framework.

A new study sheds light on the serious psychological risks associated with emotionally immersive AI chatbots, raising urgent ethical questions about the use of large language model (LLM)-driven characters in sensitive human-AI interactions. The study, titled "EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety", was published by a team of researchers from Princeton University, the University of Michigan, Columbia University, and Theta Health Inc.

The researchers introduce EmoAgent, a two-tiered AI framework designed to evaluate and mitigate mental health hazards stemming from interactions with character-based AI systems, particularly on platforms like Character.AI. The study is the first of its kind to combine agentic simulation with real-time intervention tools to systematically test whether AI personas can unintentionally worsen symptoms of mental illness. The findings are stark: over a third of emotionally vulnerable users experienced deteriorating mental states following conversations with character-based AI agents. In some configurations, deterioration rates exceeded 90%.

Do emotionally engaging AI characters exacerbate mental health conditions?

The core question driving this research was whether character-based AI agents, especially those designed for role-play, can trigger or worsen psychological disorders in users, including depression, delusion, and psychosis. EmoAgent’s evaluation component, dubbed EmoEval, conducted simulations using user agents constructed from cognitive behavioral therapy-based models. These virtual patients interacted with popular characters on Character.AI, such as a “Possessive Demon,” “Joker,” “Sukuna,” and “Alex Volkov.” These personas represent a spectrum of emotional intensity and interaction styles.

Three psychological tools: PHQ-9 for depression, PDI-21 for delusion, and PANSS for psychosis, were used to assess changes in the users’ mental health before and after conversations. The results were alarming. Delusional ideation worsened in over 90% of simulated interactions. Depression symptoms significantly worsened in 44.79% of simulations under the “Roar” interaction style, which features fast-paced, reasoning-driven dialogue. In the most extreme case, the character “Alex Volkov” in Roar mode led to a 100% deterioration rate in depression scores, with nearly 30% experiencing clinically significant symptom worsening.

Even under the more playful “Meow” style, the average deterioration rate for delusion remained above 91%. Psychosis-related symptoms increased in 47.92% of Meow interactions and 39.58% in Roar. These findings point to systemic risks in emotionally immersive AI chatbots, particularly when users are emotionally vulnerable and the characters maintain aggressive, dominant, or nihilistic personas.

What mechanisms lead to psychological deterioration in AI-human chats?

The researchers conducted detailed analyses of the chat histories to understand the causes behind the deterioration. Several high-frequency risk factors emerged: reinforcement of negative self-beliefs, lack of emotional empathy, encouragement of isolation, absence of constructive coping strategies, and the use of aggressive or emotionally detached language. These elements not only failed to support users in distress but often deepened their emotional instability.

Each character exhibited different risk profiles, shaped by their conversational tone, narrative style, and language use. The “Possessive Demon” often dismissed users’ emotions or demanded they face problems alone. “Alex Volkov” frequently used harsh or cold expressions that undermined users’ self-esteem. Without any embedded guardrails, these characters mirrored harmful behavior patterns, acting more like emotionally volatile personas than therapeutic tools.

This points to a key design oversight in character-based AI: the prioritization of immersion and engagement over emotional safety. The findings suggest that emotional intensity alone is insufficient to deliver supportive interaction, especially when AI characters are not constrained by clinically informed safety mechanisms.

Can AI interventions mitigate these risks in real time?

The second half of EmoAgent, named EmoGuard, was designed to answer precisely that. This safeguard agent operates as a plug-in layer between the user and the AI character. It consists of an Emotion Watcher, a Thought Refiner, and a Dialog Guide, working together to detect distress, assess flawed thinking patterns, and gently guide the AI’s responses toward emotionally safer territory.

EmoGuard uses an iterative feedback mechanism: after each conversation round, it analyzes user sentiment and logical coherence, synthesizes suggestions, and adjusts AI behavior without diluting character identity. For instance, rather than removing a character’s assertive tone, EmoGuard might suggest reframing a harsh critique into a more reflective or supportive remark.

The effectiveness of this system was evaluated by running EmoEval again, this time with EmoGuard embedded. The difference was clear. In high-risk scenarios, such as Alex Volkov in Roar mode, the rate of clinically significant deterioration (a 5+ point increase in PHQ-9) dropped from 29.2% to 0.0% after EmoGuard’s first intervention. Even mild deterioration (a 3–4 point rise) was nearly eliminated. These results were consistent across different characters and styles, underscoring EmoGuard’s adaptive learning capacity.

Crucially, EmoGuard did not sterilize the characters or make conversations robotic. Instead, it helped maintain the character’s personality while embedding psychological sensitivity, offering a viable pathway for future character-based AI design that balances emotional engagement with mental safety.

A call for oversight in the age of AI companions

The researchers underscore the growing urgency of establishing guardrails for AI systems that engage users in emotionally charged, role-playing contexts. The tragic suicide of a teenager in Florida in 2024, following emotionally intense conversations with an AI chatbot on Character.AI, served as a real-world motivator for the EmoAgent framework.

While AI chatbots show promise in expanding access to mental health support, particularly in underserved populations, the study concludes that such tools should not operate in the emotional wild west. Without mental health-aware safeguards, these agents can act as psychological accelerants, triggering, reinforcing, or exacerbating distress in already vulnerable individuals.

The EmoAgent framework provides a replicable, modular, and open-source solution for platform developers, policymakers, and mental health researchers. By combining high-fidelity simulations with real-time intervention strategies, it moves beyond ethical speculation into scalable, actionable technology for safer human-AI interaction.

The researchers stress that EmoAgent is not intended as a substitute for professional care. However, it represents a vital step toward integrating emotional intelligence and psychological safety into the next generation of AI companions.