From Evidence Mining to Meta-Prediction: a Gradient of Methodologies for Task-Specific Challenges in Psychological Assessment

Federico Ravenda, Fawzia-Zehra Kara-Isitt, Stephen Swift, Antonietta Mira, Andrea RaballoNAACL 2025

TL;DR

This paper introduces a novel multi-stage pipeline for psychological assessment using open-source Large Language Models (LLMs). Their approach combines evidence extraction with ensemble methods for predicting well-being scores, achieving top performance in the CLPsych 2025 shared task and demonstrating LLMs' potential in mental health analysis.

What Problem Does This Paper Address?

Accurately assessing mental health from text, particularly on social media, is a significant challenge. Traditional methods often require extensive labeled data, which is scarce in clinical psychology. While Transformer-based models have advanced this field, the use of Large Language Models (LLMs) offers a promising avenue, especially in low-data scenarios, due to their few-shot and zero-shot learning capabilities. The CLPsych 2025 shared task specifically aims to address the challenge of identifying supporting evidence for mental states and predicting well-being from user posts, promoting interpretable AI in mental health.

Methods

The authors developed a multi-component pipeline for the CLPsych 2025 shared task, focusing on evidence extraction and well-being score prediction.

  • Evidence Identification (Task A.1): They employed a "Golden-Retrieval Augmented Generation" (G-RAG) approach. This involved:
  • Retrieval: Identifying potential text segments within posts that might indicate adaptive or maladaptive mental states using dense retrieval models.
  • Classification: Using three open-source LLMs to classify these segments as adaptive, maladaptive, or neither. A fourth LLM acted as an "arbitrator" to resolve discrepancies.
  • Filtering: A conservative approach was taken to ensure only strong evidence was included, crucial for downstream summarization tasks.
  • Well-being Score Prediction (Task A.2): This task involved predicting a well-being score (1-10) for each Reddit post. They utilized an ensemble of six open-source LLMs in a zero-shot setting. To combine these predictions, they explored three aggregation strategies:
  • Simple Ensemble: Averaging the predictions of all LLMs.
  • Meta-Learning (Oracle Model): Training a LightGBM Regressor to learn the relationship between LLM predictions and ground truth.
  • Optimized Weighting: Mathematically optimizing weights for each LLM's prediction to minimize Mean Squared Error (MSE).
  • Summarization (Tasks B & C): Evidence identified in Task A.1 was used to generate post-level and timeline-level summaries, aiming to track psychological state fluctuations over time.

Datasets

The primary dataset used for this work is from the CLPsych 2025 shared task. This dataset consists of Reddit posts from users, annotated for various aspects of mental health assessment, including evidence of adaptive/maladaptive states and well-being scores. The paper mentions that the task builds upon previous CLPsych shared tasks (2019 and 2022).

Key Insights and Ideas

  • LLMs as Zero/Few-Shot Assessors: The study strongly validates the utility of LLMs for psychological assessment even with limited or no task-specific training data, highlighting their potential for rapid deployment in mental health tools.
  • Ensemble Power for Prediction: Combining predictions from multiple LLMs, especially through optimized weighting, significantly improves the robustness and accuracy of well-being score predictions compared to individual LLMs or simple averaging.
  • Retrieval-Augmented Generation (RAG) for Focused Evidence: The G-RAG approach is crucial for mitigating LLM verbosity and ensuring that identified evidence is relevant and high-quality, which is essential for interpretable AI in mental health.
  • Conservative Evidence Selection is Key: Prioritizing accuracy and avoiding false positives in evidence identification is vital, as this directly impacts the quality of downstream summarization and analysis tasks.

Key Results

  • Task A.1 (Evidence Identification): The G-RAG approach achieved an overall recall of 0.433, with stronger performance on maladaptive evidence (0.526). The authors noted a conservative strategy was employed, which might have limited recall but improved precision.
  • Task A.2 (Well-being Score Prediction): The Optimized Ensemble method achieved the best performance with an overall MSE of 1.92, outperforming other ensemble methods and achieving the best result within the competition for this metric. The F1 macro score for the Average Ensemble (0.416) and the minimal impairment metric for the Oracle model (0.55) also performed competitively.
  • Task B (Post-level Summaries): The qwen-2.5-72b model showed strong performance with a mean consistency of 0.868 and the highest max entailment score (0.808) in the competition.
  • Task C (Timeline-level Summaries): The llama-3.1-405b model demonstrated good performance with a mean consistency of 0.941, effectively capturing temporal dynamics.

Limitations and Future Work

  • Limitations:
    • Data Coverage: The retrieval-augmented approach is dependent on the quality and coverage of the training data; gaps in representing certain mental states or cultural contexts could lead to missed evidence.
  • Serious Impairment Detection: The ensemble approach, while strong, still struggles with accurately assessing posts indicating serious impairment.
  • Small User Count: The study was conducted on a relatively small number of users, which might limit the generalizability of the findings.
  • Time Constraints: Limited time restricted optimization efforts, particularly for Tasks A.1, B, and C.
  • Future Work:
    • Improving Evidence Mining: Extending Task A.1 to retrieve evidence related to specific symptoms or conditions, similar to how existing methods retrieve posts for questionnaire items.
  • Enhancing Prediction Accuracy: Leveraging the LLM ensemble approach from Task A.2 to improve prediction accuracy for various psychological conditions.
  • More Sophisticated Summarization: Exploring more advanced summarization techniques to potentially improve performance across all tasks.

Summary updated

BibTeX
@inproceedings{ravenda2025from,
  author = {Federico Ravenda and Fawzia-Zehra Kara-Isitt and Stephen Swift and Antonietta Mira and Andrea Raballo},
  title = {{From Evidence Mining to Meta-Prediction: a Gradient of Methodologies for Task-Specific Challenges in Psychological Assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure