Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case

Kaushik Roy, Harshul Surana, Darssan Eswaramoorthi, Yuxin Zi, Vedant Palit, Ritvik Garimella, Amit Sheth

Abstract

Large language models (LLMs) are increasingly attracting the attention of healthcare professionals for their potential to assist in diagnostic assessments, which could alleviate the strain on the healthcare system caused by a high patient load and a shortage of providers. For LLMs to be effective in supporting diagnostic assessments, it is essential that they closely replicate the standard diagnostic procedures used by clinicians. In this paper, we specifically examine the diagnostic assessment processes described in the Patient Health Questionnaire-9 (PHQ-9) for major depressive disorder (MDD) and the Generalized Anxiety Disorder-7 (GAD-7) questionnaire for generalized anxiety disorder (GAD). We investigate various prompting and fine-tuning techniques to guide both proprietary and open-source LLMs in adhering to these processes, and we evaluate the agreement between LLM-generated diagnostic outcomes and expert-validated ground truth. For fine-tuning, we utilize the Mentalllama and Llama models, while for prompting, we experiment with proprietary models like GPT-3.5 and GPT-4o, as well as open-source models such as llama-3.1-8b and mixtral-8x7b.

Introduction

The global healthcare system faces a critical shortage of mental health providers and an overwhelming patient load. While Large Language Models (LLMs) are often proposed as a solution, their utility depends on their ability to follow standardized clinical procedures. Most previous research treated mental health detection as a "black box" classification task (e.g., "Does this person have depression? Yes/No"). However, clinicians do not work that way; they use validated instruments like the Patient Health Questionnaire-9 (PHQ-9) and the Generalized Anxiety Disorder-7 (GAD-7) to identify specific symptoms.

Standardized clinical instruments: PHQ-9 for depression and GAD-7 for anxiety.

The gap this paper addresses is whether LLMs can move beyond simple classification to identify the specific "evidence" (text spans) in a patient's language that corresponds to these clinical criteria. By mapping social media text to these standardized tools, we explore how LLMs can assist in screenings while maintaining clinical interpretability.

Methodology

The study explored two primary strategies to align LLMs with clinical workflows: prompting and fine-tuning.

Prompting Strategies

Three levels of prompting were tested to evaluate model adaptability:

  1. Naive: Direct instructions without additional context.
  2. Exemplar-based: Providing few-shot examples of the desired output to guide the model.
  3. Guidance-based: Using Chain-of-Thought (CoT) to steer the model through explicit reasoning steps.

Fine-tuning and DiagnosticLlama

The researchers developed DiagnosticLlama, a specialized model fine-tuned using the Llama and MentalLLaMA architectures on the PRIMATE dataset. This involved specialized "steering" to ensure the model outputs information relevant to specific PHQ-9 and GAD-7 symptoms, rather than general sentiment.

A completed GAD-7 assessment, illustrating the symptom-based scoring system LLMs aim to replicate.

Datasets and Validation

  • PRIMATE Dataset: A public collection of social media posts annotated for PHQ-9 and GAD-7 criteria.
  • Expert Validation: Three experts from the National Institute of Mental Health and Neurosciences (NIMHANS) validated a subset of GPT-4o annotations, achieving a Cohen’s Kappa of 0.74, establishing a high-quality "gold standard."
  • Evaluation Metrics: Beyond standard accuracy, the study used hits@k based ranking to measure if model-identified text spans (ranked by cosine similarity to a symptom) appeared in the expert-validated ground truth.

Key Insights and Results

The Reasoning Gap

A striking finding was that while LLMs can match a clinician's outcome, they only match a clinician's reasoning a fraction of the time. This suggests that LLMs might arrive at the correct diagnostic conclusion through different linguistic patterns than those used by human experts.

Performance Benchmarks

  • Top Performers: GPT-4o-mini and Mixtral-8x7b were the standout models. For PHQ-9 symptom annotation, GPT-4o-mini achieved 0.94 accuracy and a 0.92 F1-score.
  • Open-Source Parity: Mixtral-8x7b achieved an impressive 0.93 F1-score on PHQ-9 tasks, matching or exceeding proprietary models. In some anxiety assessment tasks, Mixtral actually outperformed GPT-4o in F1-score.
  • DiagnosticLlama Success: The specialized fine-tuned model achieved 68.3% hits@1, demonstrating that smaller, specialized models can effectively identify symptom-relevant text.
  • Zero-shot Struggles: LLMs generally struggle to utilize questionnaire information effectively in a zero-shot setting. Performance improves significantly once few-shot examples or fine-tuning are introduced.

Conclusion and Future Work

Modern LLMs significantly outperform older, encoder-only models like BERT and MentalBERT, proving that generative capabilities provide a superior understanding of clinical nuance. However, challenges remain regarding the explainability of model reasoning.

Future work will focus on expanding to more complex, non-linear diagnostic tools, such as the Columbia-Suicide Severity Rating Scale (C-SSRS). The ultimate goal is the integration of these models into clinician-facing applications that include robust reasoning explanations and guardrails for non-toxic terminology.

Summary updated

Abstract

Large language models (LLMs) are increasingly attracting the attention of healthcare professionals for their potential to assist in diagnostic assessments, which could alleviate the strain on the healthcare system caused by a high patient load and a shortage of providers. For LLMs to be effective in supporting diagnostic assessments, it is essential that they closely replicate the standard diagnostic procedures used by clinicians. In this paper, we specifically examine the diagnostic assessment processes described in the Patient Health Questionnaire-9 (PHQ-9) for major depressive disorder (MDD) and the Generalized Anxiety Disorder-7 (GAD-7) questionnaire for generalized anxiety disorder (GAD). We investigate various prompting and fine-tuning techniques to guide both proprietary and open-source LLMs in adhering to these processes, and we evaluate the agreement between LLM-generated diagnostic outcomes and expert-validated ground truth. For fine-tuning, we utilize the Mentalllama and Llama models, while for prompting, we experiment with proprietary models like GPT-3.5 and GPT-4o, as well as open-source models such as llama-3.1-8b and mixtral-8x7b.
BibTeX
@misc{roy2025large,
  author = {Kaushik Roy and Harshul Surana and Darssan Eswaramoorthi and Yuxin Zi and Vedant Palit and Ritvik Garimella and Amit Sheth},
  title = {{Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case}},
  year = {2025},
  eprint = {2501.01305},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.01305},
}
Figure