Mental health support tools using AI struggle with privacy and accuracy

One major concern was inaccurate or inapplicable informational support. Participants recounted instances where chatbots provided misleading medical information or advice that failed to account for physical disabilities or co-morbid conditions, potentially worsening their mental health situations. The risk of emotional damage from incorrect or irrelevant advice was flagged as an immediate design flaw.

The growing interest in artificial intelligence for mental health support is now facing fresh scrutiny as new evidence suggests AI-powered chatbots may introduce subtle yet significant risks for individuals living with depression. A recent study titled "AI Chatbots for Mental Health: Values and Harms from Lived Experiences of Depression", published on arXiv, offers a comprehensive exploration of how individuals with depression perceive the benefits and dangers of interacting with AI chatbots like GPT-4o. Researchers from Indiana University Indianapolis and the University of Illinois Urbana-Champaign developed a technology probe called "Zenny" and engaged 17 individuals in interviews, uncovering critical insights into user values, potential harms, and ethical design challenges.

What values do people with depression prioritize when using AI chatbots?

The research aimed to explore the fundamental question of which values individuals with lived experiences of depression prioritize when using AI chatbots for self-management. Through extensive scenario-based interviews, the study identified five core values: informational support, emotional support, personalization, privacy, and crisis management.

Participants consistently emphasized the need for accurate and actionable informational support, reflecting a reliance on the chatbot for guidance on everyday depression management strategies. Emotional support emerged as equally vital, with many users appreciating the chatbot’s empathetic tone and validation, although most still preferred human interaction for deeper emotional connection. Personalization of responses was heavily valued, with users seeking tailored advice based on their individual circumstances rather than generic suggestions. However, a strong thread of concern regarding privacy ran through the interviews, particularly fears about sensitive mental health data being misused if chatbots lacked robust protections. Finally, crisis management surfaced as a critical expectation: users voiced serious worries about AI chatbots' inability to properly handle emergencies like suicidal ideation or severe emotional breakdowns.

The study highlighted that users' priorities are not abstract or universal but deeply contextual, rooted in lived daily realities of managing a stigmatized and complex condition. These values, derived from real experiences rather than hypothetical use cases, form the foundation for understanding the specific risks chatbots may introduce.

What potential harms might arise from AI chatbots in mental health?

The study’s findings painted a detailed and often troubling picture of potential harms when AI chatbots are used for mental health self-management without adequate safeguards.

One major concern was inaccurate or inapplicable informational support. Participants recounted instances where chatbots provided misleading medical information or advice that failed to account for physical disabilities or co-morbid conditions, potentially worsening their mental health situations. The risk of emotional damage from incorrect or irrelevant advice was flagged as an immediate design flaw.

Emotional support offered by chatbots, while helpful in the moment, raised alarms about over-reliance on machines for emotional needs. Participants worried that heavy chatbot usage could amplify social isolation, a critical factor that exacerbates depression. Although users appreciated the judgment-free environment, they recognized that genuine human empathy was irreplaceable.

A deeper, more systemic risk emerged through what researchers termed the personalization-privacy dilemma. Users sought tailored support but were wary of sharing intimate personal data with AI systems. Many described self-protective behaviors like using incognito modes, masking personal details in queries, or maintaining separate email identities. The need for hyper-personalized advice directly conflicted with strong instincts to safeguard personal privacy, especially amid rising concerns over data breaches and misuse by corporations.

Finally, there was an overwhelming consensus that chatbots are not equipped to handle crisis scenarios. Participants stressed that without proper crisis detection and intervention protocols, AI tools could provide false reassurance during a mental health emergency, leading to tragic consequences. The risk was seen not just in poor advice but in creating a false sense of safety in critical moments.

How can AI chatbots be designed responsibly for mental health support?

The study doesn’t stop at raising alarms - it offers actionable recommendations for researchers, developers, and policymakers who are shaping the future of AI in healthcare.

For informational support, researchers recommend that chatbots clearly disclose their informational limits, urging users to cross-check medical advice and consult human professionals. Suggestions should be adapted through dynamic follow-up questions that gather necessary context from users before providing advice.

To address concerns around emotional support, chatbots should encourage users to maintain human social ties and frame themselves as complementary tools rather than replacements for therapists or real-world support networks. Features that nudge users toward human connection could mitigate social isolation risks.

In resolving the personalization-privacy dilemma, the study advocates for user-controlled data models where individuals can see, manage, and delete information collected by the chatbot. Transparent communication about what is stored, inferred, and how it’s used should become industry standard.

For crisis management, future chatbots must embed clear disclaimers about their limitations in crisis situations, integrate basic crisis response protocols, and provide seamless redirects to professional help lines and emergency services when certain risk signals are detected. Governance mechanisms should be built into chatbot ecosystems to enforce safety standards in sensitive mental health applications.