TL;DR
This paper tackles the challenge of understanding mental health shifts over time by analyzing longitudinal social media data. The authors developed a system using transformer models to extract evidence of adaptive and maladaptive self-states, predict well-being scores, and generate summaries of these dynamics at both post and timeline levels, achieving competitive results in the CLPsych 2025 shared task.
What Problem Does This Paper Address?
The paper addresses the growing need to understand and monitor mental health through digital footprints, specifically social media. The core challenge is to move beyond static snapshots of mental state and instead capture the dynamic evolution of a user's psychological well-being over time. This involves identifying subtle shifts between "adaptive" self-states (those that support fundamental needs like connection and competence) and "maladaptive" self-states (those that hinder these needs, like anxiety or hopelessness). Existing research often focuses on specific events or single posts, but this work aims to build a more comprehensive picture by analyzing longitudinal data and generating insights at different granularities – from individual posts to entire timelines. The CLPsych 2025 shared task, which this paper participates in, specifically focuses on this longitudinal analysis and evidence generation.
Methods
The authors employed a multi-task approach, leveraging various pre-trained transformer models fine-tuned for specific tasks:
- Task 1a: Evidence Extraction: To identify specific text spans indicating adaptive or maladaptive self-states, they initially considered BERT but found it prone to fragmentation. They switched to Mixtral 8x7B, a large language model known for its robust natural language understanding, using a carefully designed prompt and a "explain then annotate" strategy to improve accuracy.
- Task 1b: Well-Being Score Prediction: For predicting a well-being score for each post, they used embeddings from various transformer models (including MentalBERT, BERT, MiniLM, ROBERTa, and ClinicalBERT) as input features for a Random Forest Classifier. MentalBERT embeddings yielded the best performance, likely due to its domain-specific pre-training.
- Task 2: Post-Level Summaries: To summarize the interplay between adaptive and maladaptive self-states within a single post, they initially tried BART but found its context window too short. They opted for Longformer, which excels at handling longer sequences, using extracted evidence spans from Task 1a as additional input.
- Task 3: Timeline-Level Summaries: For generating summaries of self-state dynamics across an entire timeline, they used Pegasus-X-Large. This model was chosen for its ability to process very long sequences, concatenating posts from the same timeline while preserving chronological order.
Datasets
The paper utilizes the dataset provided for the CLPsych 2025 Shared Task. The initial dataset contained 343 rows. For Tasks 1b and 3, the dataset was reduced to 199 rows after removing null values in well-being scores and post summaries. Tasks 1a and 2 used the full dataset. The paper does not introduce a new benchmark dataset but rather participates in an existing shared task.
Key Insights and Ideas
- Domain-Specific Models Shine: The superior performance of MentalBERT for well-being score prediction highlights the advantage of models pre-trained on domain-specific data (mental health texts) for capturing nuanced psychological signals.
- Long-Context Models are Crucial: The switch from BART to Longformer for post-level summarization and the choice of Pegasus-X-Large for timeline summarization underscore the necessity of models capable of handling extended text sequences for longitudinal analysis.
- Mixtral 8x7B for Precise Extraction: Using a powerful MoE model like Mixtral 8x7B with careful prompting proved effective for the challenging task of precisely extracting evidence spans, avoiding fragmentation issues.
- "Explain then Annotate" Boosts Accuracy: The integration of an "explain then annotate" strategy with Mixtral 8x7B improved the accuracy of span extraction by forcing deeper contextual understanding before classification.
- Traditional Models Still Competitive: Despite the rise of prompt-tuned LLMs, the authors demonstrate that fine-tuning traditional transformer models like BERT, Longformer, and Pegasus can still yield competitive results, especially when tailored to specific tasks.
Key Results
The team, "ResBin," achieved an overall ranking of 13th in the CLPsych 2025 shared task, with specific rankings:
- Task 1a (Evidence Extraction): 8th
- Task 1b (Well-Being Score Prediction): 13th
- Task 2 (Post-Level Summaries): 8th
- Task 3 (Timeline-Level Summaries): 7th
Qualitatively, they report that MentalBERT achieved state-of-the-art performance in well-being score prediction, Longformer provided detailed post summaries, Pegasus-X effectively distilled timeline insights, and Mistral showed high recall in evidence extraction.
Limitations and Future Work
The authors acknowledge several limitations:
- Traditional vs. Prompt-Tuned LLMs: Using traditional language models rather than prompt-tuned LLMs might have limited performance on tasks requiring highly nuanced psychological inferences.
- Limited Data & Augmentation: The relatively small dataset size and lack of data augmentation meant that pre-trained models couldn't generalize as extensively as LLMs might, and fine-tuning LLMs was not explored.
- Long-Term Dependencies: Their models might inadequately capture long-term temporal dependencies, potentially missing subtle shifts in mental states.
Future work could explore supervised fine-tuning of large language models (LLMs) with sufficient computational resources to further enhance prediction and summarization capabilities, potentially bridging the gap between general-purpose models and domain-specific mental health analysis.
Summary updated