Abstract
Automated mental health analysis shows great potential for enhancing the efficiency and accessibility of mental health care, with recent methods using pre-trained language models (PLMs) and incorporated emotional information. The latest large language models (LLMs), such as ChatGPT, exhibit dramatic capabilities on diverse natural language processing tasks. However, existing studies on ChatGPT for mental health analysis bear limitations in inadequate evaluations, ignorance of emotional information, and lack of explainability. To bridge these gaps, we comprehensively evaluate the mental health analysis and emotional reasoning ability of ChatGPT on 11 datasets across 5 tasks, and analyze the effects of various emotion-based prompting strategies. Based on these prompts, we further explore LLMs for interpretable mental health analysis by instructing them to also generate explanations for each of their decisions. With an annotation protocol designed by domain experts, we convey human evaluations to assess the quality of explanations generated by ChatGPT and GPT-3. Experimental results show that ChatGPT outperforms traditional neural network-based methods but still has a significant gap with advanced task-specific methods. Prompt engineering with emotional cues can be effective in improving performance on mental health analysis but suffers from a lack of robustness and inaccurate reasoning. In addition, ChatGPT significantly outperforms GPT-3 on all criteria in human evaluations of the explanations and approaches to human performance, showing its great potential in explainable mental health analysis.
Introduction
Mental health analysis is a "safety-critical" domain where automated systems require high transparency and reliability. While Large Language Models (LLMs) have shown general linguistic prowess, their application in mental health has been hindered by three main gaps: a lack of interpretability in binary classifications, inadequate prompting that ignores emotional cues used by human experts, and a scarcity of rigorous benchmarks comparing LLM-generated explanations against expert standards.
Methodology
The researchers evaluated four LLMs: ChatGPT (gpt-3.5-turbo), InstructGPT-3, LLaMA-13B, and LLaMA-7B. The study moved beyond simple "zero-shot" prompts by testing several advanced strategies:
- Chain-of-Thought (CoT): Instructing the model to "think step-by-step."
- Emotion-enhanced CoT: A novel strategy where the prompt explicitly directs the model to identify emotional cues (using lexicons like VADER and NRC EmoLex) before making a decision.
- Few-shot Learning: Providing the model with a few expert-written examples of posts and their corresponding explanations.
To evaluate explainability, a human study was conducted where PhD students in psychology rated model explanations based on Fluency, Reliability, and Completeness.
Datasets
The study utilized 11 datasets covering five distinct tasks:
- Mental Health Detection: DR (Reddit depression), CLPsych15, and Dreaddit (stress).
- Cause/Factor Detection: SAD (stressors) and CAMS (reasons for depression/suicide).
- Emotional Reasoning: T-SID (suicide ideation), IEMOCAP, MELD, EmoryNLP, DailyDialog, and RECCON.
- New Benchmark: The authors created a novel dataset of 163 human-assessed explanations to serve as a "gold standard" for future research.
Key Results
Experimental results indicate that while ChatGPT is a strong performer among LLMs, it still lags behind specialized, fine-tuned models like MentalRoBERTa. For instance, on the CAMS dataset, MentalRoBERTa achieved a 47.62 F1 score, while ChatGPT's best prompted version reached 45.99.
In human evaluations, ChatGPT's explanations for correct classifications received an average score of 2.5/3.0, approaching human performance and significantly outperforming InstructGPT-3. Additionally, the authors found that BART-Score correlated most strongly with human judgments, suggesting it is a superior tool for automatically evaluating mental health explanations compared to traditional metrics like BLEU.
Discussion and Insights
- The "Emotion Boost": Standard CoT is often ineffective for mental health tasks. However, explicitly prompting models to look for emotions (Emotion-enhanced CoT) significantly improves accuracy, suggesting LLMs require specific "triggers" to apply latent knowledge to clinical contexts.
- Explainability vs. Accuracy: ChatGPT generates highly fluent explanations for correct classifications. However, when the model makes an error, its explanation remains fluent but becomes unreliable, often hallucinating or misinterpreting the context.
- The Sensitivity Trap: LLMs exhibit significant instability in this domain. Minor changes in prompt adjectives (e.g., "any" stress vs. "very severe" stress) caused performance variance of up to 89% on certain datasets.
Conclusion
While ChatGPT shows great potential for explainable mental health analysis, it currently suffers from inaccurate reasoning in complex cases and a lack of deep clinical knowledge. The models occasionally struggle with context, such as misattributing quoted distress to the poster. Future work should focus on domain-specific fine-tuning and the development of automatic metrics tailored specifically to the nuances of mental health discourse.
Summary updated