Abstract
Analyzing mental health through social media requires both high accuracy and clinical interpretability. Traditional models often rely on massive labeled datasets or produce "black-box" explanations that lack scientific grounding. This paper introduces MAIMS (Mental Analysis by Incorporating Mental Scales), a zero-shot framework that mimics clinical psychological practice. By requiring Large Language Models (LLMs) to complete standardized mental health scales before rendering a diagnosis, MAIMS achieves state-of-the-art zero-shot performance and even outperforms specialized supervised models in specific contexts.
Introduction
Social media has become a vital resource for early mental health intervention, yet automated analysis faces significant hurdles. Most existing models are either discriminative—requiring expensive expert-labeled data—or generative, which can lead to "hallucinated" symptoms or explanations that lack a scientific basis. Unlike human psychologists who utilize validated tools like the Beck Depression Inventory (BDI), many AI systems jump directly from raw text to a diagnosis.
To address these challenges, the MAIMS framework introduces a structured, two-step workflow that incorporates established psychological scales as an interpretive lens. This approach ensures that the model's reasoning is traceable, grounded in clinical standards, and capable of operating without task-specific training data.
The MAIMS Framework
MAIMS utilizes a multi-agent approach involving three distinct LLM roles to simulate a clinical environment:
- Step 1: Mental Scale Completion ($L_{poster}$): An LLM acts as a proxy for the patient. It analyzes the social media post and attempts to answer questions from a standardized mental health scale (e.g., the BDI or INQ-15). This step translates informal language into structured clinical indicators.
- Step 2: Mental Health Analysis ($L_{analysis}$): A second LLM acts as the psychologist. It reviews both the original post and the completed scale to provide a final assessment and a reasoned explanation.
- The Discriminator ($L_{discriminator}$): To prevent the model from over-inferring or hallucinating symptoms, a discriminator model performs a "sanity check" at each stage. If the scale completion or the final analysis is logically inconsistent with the source text, the process is repeated.
This zero-shot methodology relies on the inherent reasoning capabilities of LLMs and the structural rigor of clinical scales rather than labeled training sets.
Methodology and Datasets
The effectiveness of MAIMS was evaluated using two primary benchmarks:
- Depression_Reddit (DR): This task utilized the Beck Depression Inventory (BDI), a 21-question inventory used to measure the severity of depression.
- IRF Dataset: This task utilized the Interpersonal Needs Questionnaire (INQ-15), which focuses on psychological factors such as "thwarted belongingness" and "perceived burdensomeness."
Key Insights
LLMs as Proxy Patients
The study demonstrates that LLMs are remarkably proficient at inferring how a user might respond to a clinical survey based on informal writing. By using these scales as "anchors," the model's final explanation becomes traceable. Instead of a generic classification, the model can point to specific scale responses—such as persistent sadness or loss of interest—to support its diagnosis.
The Role of the Discriminator
Ablation studies highlighted the necessity of the Discriminator component. LLMs often exhibit a bias toward being "too empathetic," frequently over-interpreting vague expressions of sadness as clinical symptoms. The discriminator acts as a rigorous filter, ensuring that every diagnostic claim is backed by evidence present in the text.
Results
MAIMS demonstrated superior performance across multiple metrics:
- Zero-Shot Performance: On the Depression_Reddit task, MAIMS achieved a Weighted F1-score of 88.68, significantly higher than raw GPT-3.5 (86.66) and ChatGPT (82.41).
- Outperforming Supervised Models: On the IRF dataset, MAIMS (77.93 F1) outperformed specialized, fine-tuned models including MentalBERT (76.73) and RoBERTa-base (71.35), despite having no prior exposure to the dataset.
- Ablation Impact: Removing the mental scales from the workflow resulted in an 11-point drop in the F1-score on the DR dataset, confirming that the clinical scale step is the primary driver of the framework's accuracy.
Limitations and Future Work
While MAIMS sets a new standard for zero-shot analysis, it currently trails behind the most advanced supervised discriminative models on certain datasets. Additionally, the multi-step process (Scale completion → Discrimination → Analysis → Discrimination) increases computational costs and latency compared to single-pass models. Future research will focus on expanding the framework to include scales for other conditions, such as anxiety, PTSD, and bipolar disorder, to broaden its clinical utility.
Conclusion
MAIMS provides a roadmap for developing AI mental health tools that are more "clinician-like." By moving away from simple black-box classification and toward evidence-based, structured reasoning, this framework enhances both the accuracy and the trustworthiness of automated mental health assessments.
Summary updated