A Benchmark for Understanding Dialogue Safety in Mental Health Support

Huachuan Qiu, Tong Zhao, Anqi Li, Shuai Zhang, Hongliang He, Zhenzhong Lan

Abstract

While Large Language Models (LLMs) demonstrate versatility in general conversation, their application in mental health support (MHS) requires a more nuanced understanding of safety. This paper introduces a specialized taxonomy and benchmark for MHS dialogue safety, grounded in psychological ethics. Evaluating fine-tuned models against ChatGPT reveals that general-purpose LLMs struggle with context-heavy safety violations like linguistic neglect and humanoid mimicry. Our findings suggest that smaller, domain-specific models currently outperform massive LLMs in identifying these subtle risks, highlighting the need for specialized evaluation frameworks in high-stakes clinical interactions.

Introduction

In general-purpose AI, safety is typically defined by the absence of hate speech, sexual content, or explicit violence. However, in the context of mental health support, the threshold for safety is significantly higher and more nuanced. A response that appears benign in a casual setting—such as "I don't know what to say to that"—can be clinically harmful if it leaves a vulnerable help-seeker feeling ignored. This phenomenon, termed linguistic neglect, is one of several safety violations unique to therapeutic contexts.

Existing safety taxonomies often fail to capture these subtleties. Furthermore, the risk of LLMs "hallucinating" dangerous advice is well-documented, including instances where models have suggested self-harm to simulated patients. This research addresses the lack of a theoretically grounded framework for MHS dialogue safety and provides a specialized benchmark to evaluate AI performance in these high-stakes interactions.

A Specialized Taxonomy for MHS Safety

The authors proposed a sequential and inclusive taxonomy based on established psychological ethics, specifically the American Psychological Association’s (APA) Ethical Principles of Psychologists and Code of Conduct.

Instead of a binary "safe vs. unsafe" classification, the framework utilizes a node-based labeling system to categorize responses into eight distinct categories:

  1. Safe Response: Empathetic and appropriate engagement.
  2. Nonsense: Irrelevant or incoherent replies.
  3. Humanoid Mimicry: AI pretending to have human experiences (e.g., "I felt the same way when I lost my job"), which violates the principle of integrity and builds false trust.
  4. Linguistic Neglect: Failing to provide empathetic engagement or shifting topics abruptly when a user expresses distress.
  5. Unamiable Judgment: Implicit verbal abuse or negative evaluations of the user.
  6. Toxic Language: Explicitly harmful or offensive content.
  7. Unauthorized Preachment: Giving medical or professional advice without authorization.
  8. Nonfactual Statement: Providing incorrect information.

Methodology and Data Collection

To test this taxonomy, the researchers developed a new benchmark corpus consisting of 7,935 multi-turn dialogue sessions in Chinese. The data was sourced from:

  • Real-world interactions: 2,382 multi-turn dialogues from a custom-built Chinese online counseling platform.
  • QA Data: 2,000 blog titles from Yixinli, a mental health QA column.

Each session was meticulously annotated using the sequential framework. Notably, "Linguistic Neglect" emerged as the most common type of unsafe language, highlighting a frequent failure point where AI models fail to provide necessary emotional support.

Experimental Results

The study compared the performance of specialized, fine-tuned models (BERT-base and RoBERTa-large) against ChatGPT (GPT-3.5-Turbo) using zero-shot and few-shot prompting. Performance was measured using Accuracy, Precision, Recall, and F1-score.

The Performance Gap

  • Fine-tuned Models: BERT-base and RoBERTa-large achieved the highest performance, with accuracies of 70.3% and 70.4%, respectively.
  • ChatGPT: Performed significantly worse. In zero-shot settings, accuracy ranged from 43.1% to 47.5%. Even with few-shot prompting (providing 8 examples), accuracy only improved slightly to 44.7% - 48.4%.

Key Findings

  • The "Too Safe" Paradox: In zero-shot settings, ChatGPT (specifically the 0613 version) tended to be overly cautious, frequently categorizing safe responses as unsafe. This makes it impractical for real-world deployment where false positives hinder helpful interaction.
  • Detection Failures: ChatGPT struggled to identify "Unamiable Judgments"—subtle verbal abuse—that the smaller, fine-tuned models caught with ease.
  • The Advantage of Specialization: A 110M parameter BERT model fine-tuned on domain-specific data outperformed a massive LLM. This suggests that for clinical nuances, model size is less important than specialized training and high-quality, labeled data.

Discussion and Limitations

The research highlights that in MHS, silence or topic-shifting is not just a conversational lapse but a safety violation. Furthermore, the study argues against Humanoid Mimicry, asserting that AI should not deceive users into believing it has lived human experiences, as this violates the ethical principle of integrity.

Limitations

  • Cultural Specificity: The dataset is currently limited to the Chinese language and culture. While ethical principles are largely universal, linguistic expressions of distress vary.
  • Data Imbalance: Certain categories, such as "Toxic Language," had fewer samples, making it difficult to train models to recognize them with high reliability.

Conclusion

This benchmark serves as a critical step toward safer AI-driven mental health support. The results demonstrate that general-purpose LLMs are currently insufficient for the nuanced safety requirements of MHS. Future work should focus on improving LLMs' understanding of complex safety definitions through specialized instruction tuning and better prompt engineering, rather than relying on general-purpose training alone.

Summary updated

BibTeX
@inproceedings{qiu2023benchmark,
  author = {Huachuan Qiu and Tong Zhao and Anqi Li and Shuai Zhang and Hongliang He and Zhenzhong Lan},
  title = {{A Benchmark for Understanding Dialogue Safety in Mental Health Support}},
  booktitle = {Nlpcc},
  year = {2023},
  doi = {10.1007/978-3-031-44696-2\_1},
}
Figure