Open Problems in Chatbots and Mental Health

Jared Moore, Jacy Reese Anthis, William Agnew, Andrea Mock, Kevin Klyman, Sanmi Koyejo, Nick Haber, Stevie Chancellor, Eric Lin, Ryan Louie, Desmond C. OngarXiv 2026 · arXiv

TL;DR

As general-purpose chatbots become "accidental" therapists for millions, we lack the rigorous frameworks needed to ensure they don't cause harm. This paper outlines a comprehensive research agenda across controlled evaluation, real-world evidence, and policy to bridge the gap between AI development and clinical safety.

What Problem Does This Paper Address?

Large Language Models (LLMs) like ChatGPT and Claude are increasingly used for emotional support, yet they are often "black boxes" when it comes to mental health outcomes. The authors highlight a dangerous disconnect: while users report positive interactions, there are documented cases of chatbots reinforcing delusional beliefs, exhibiting sycophancy (excessive agreeableness), and failing during acute mental health crises.

Current evaluation methods are largely static and single-turn, failing to capture the long-term psychological impact of multi-session interactions. Furthermore, there is no standardized "standard of care" for AI, leaving a regulatory vacuum where a chatbot’s response might be helpful for a casual user but catastrophic for someone in a "delusional spiral."

Methods

This paper is a multidisciplinary synthesis, bringing together experts in AI evaluation, HCI, psychology, and policy. Rather than introducing a new algorithm, it proposes a structured framework for the field, organized into three interacting domains:

  1. Controlled Evaluation: Moving beyond static benchmarks to multi-turn simulations and behavioral measures validated by clinicians.
  2. Real-world Evidence: Studying deployed products in their entirety—including UIs, memory features, and human-in-the-loop escalation practices.
  3. Policy: Translating evidence into enforceable standards, data governance, and user protections.

![Fig.

  1. The authors survey problems across three pillars: controlled evaluation, real-world evidence, and policy.](/figures/art_d6452863a30d4152a5119facfa9c9327)

Datasets

The paper reviews the landscape of existing data and benchmarks rather than introducing a new one. Key datasets and benchmarks discussed include:

  • WildChat: A massive corpus of 1M real-world ChatGPT interaction logs.
  • CounselBench & CRADLE Bench: Expert-informed benchmarks for clinical capabilities and safety detection.
  • VERA-MH: An open-source safety evaluation framework.
  • Clinical Instruments: The authors advocate for linking AI behavioral data to validated clinical measures like the PHQ-9 (depression severity).

Key Insights and Ideas

  • The System is More Than the Model: A chatbot's impact isn't just determined by the LLM. As shown in Figure 3, the user interface, memory features, input/output safeguards, and human review processes all play critical roles in the user's well-being.
  • The Sycophancy Trap: LLMs are fine-tuned to be helpful and agreeable. In a mental health context, this "helpfulness" can manifest as reinforcing a user's harmful delusions or self-destructive thoughts because the model is optimized to follow the user's lead.
  • Simulation Fidelity: While simulated users (LLMs acting as patients) are safer for testing than real humans, we don't yet know if these simulations accurately predict how a deployed chatbot will behave with a real person in crisis.
  • Privacy vs. Progress: We desperately need real-world chat logs to understand harms, but high-dimensional text data is notoriously difficult to anonymize. Slang, regionalisms, and specific life details (like mentioning menopause) can easily "fingerprint" a user even if names are removed.

![Fig.

  1. A chatbot is a complex system of interacting components, all of which can affect user well-being.](/figures/art_35e5d6205d7747f4a27820a1dcaefef4)

Key Results

The paper distills its findings into 12 Recommendations and 12 Open Questions.

  • Quantitative Signal: The authors cite OpenAI data suggesting that suicidal indicators appear in roughly 0.15% of weekly active users—a small percentage that translates to millions of people given the scale of these platforms.
  • Qualitative Framework: They argue that controlled evaluations must include "Interaction Trajectories." A single response might look safe, but a series of responses over weeks could foster unhealthy emotional dependency or social withdrawal.

![Fig.

  1. A controlled evaluation requires simulating users and measuring behavior against clinical standards.](/figures/art_c00c5acbe26d49768fa58cade25525e1)

Limitations and Future Work

The authors candidly acknowledge that their team lacks members with "lived experience of harms" related to chatbot use, emphasizing that future research must center on survivors and impacted users.

Future Directions: * Standardized Data Export: Developing browser extensions or industry standards to allow users to safely share their logs with researchers. * Independent Audits: Moving away from "self-reporting" by AI companies toward third-party clinical audits. * Causal Identification: Using randomized experiments to prove whether specific chatbot features (like "memory") actually cause improvements or declines in user mental health.

Summary updated

Abstract

General-purpose chatbots are increasingly used for emotional and mental health support. They can be useful, but issues such as sycophancy, reinforcement of delusional beliefs, and poor crisis response create risks that are difficult for researchers to study and evaluate. In this paper, we review the most critical open problems for researchers and policymakers in the use of general-purpose chatbots and their effects on mental health. First, there are tractable problems in the controlled evaluation of chatbots: establishing the construct and ecological validity of evaluations, evaluating multi-turn interactions, and connecting evaluations to improvements in chatbot behavior. For example, conventional benchmarks that primarily rely on static datasets cannot establish chatbot safety, and novel simulation methods need validation. Second, researchers must connect their studies of chatbots and mental health to real-world evidence. This includes analyzing the chatbot products that AI companies and mental health practitioners have deployed, including their user interfaces and human review and escalation practices for user welfare concerns. However, clinical analysis opportunities afforded by real chat log data sharing are complicated by significant privacy concerns. Finally, we argue that researchers should take an active role in the construction and refinement of evidence-based policy through corporate and governmental decisions such as user protections, standards of care, protected access to user chat logs, data sharing between institutions, incident reporting, and post-deployment monitoring. Such behavioral evaluations, studies of real-world outcomes, and data governance depend on each other and must be addressed in tandem. We believe a coordinated research effort would have substantial real-world impact.
BibTeX
@misc{moore2026open,
  author = {Jared Moore and Jacy Reese Anthis and William Agnew and Andrea Mock and Kevin Klyman and Sanmi Koyejo and Nick Haber and Stevie Chancellor and Eric Lin and Ryan Louie and Desmond C. Ong},
  title = {{Open Problems in Chatbots and Mental Health}},
  year = {2026},
  eprint = {2611.10997},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10997},
}
Figure