TL;DR
This paper introduces a novel approach to model mental health dynamics over time using social media posts. By employing hybrid Large Language Models (LLMs), they can identify adaptive and maladaptive emotional states within individual posts, predict overall well-being, and generate both post-level and timeline-level summaries, offering a more nuanced understanding of mental health trajectories.
What Problem Does This Paper Address?
The early detection of mental disorders is crucial, and social media offers a rich source of data for this purpose. However, existing research often focuses on analyzing individual posts in isolation, neglecting the longitudinal dynamics of an individual's mental health. Furthermore, training LLMs on sensitive mental health data is challenging due to privacy concerns and the difficulty of data collection and annotation. This paper aims to bridge this gap by developing a method to model mental health over time from social media, while also addressing the challenges of data scarcity and privacy through prompt engineering and hybrid LLM approaches.
Methods
The authors propose a hybrid LLM framework that tackles four key tasks:
- Adaptive/Maladaptive Evidence Identification: This is the foundational step. They employ two strategies:
- Classification-based: A fine-tuned LLM classifies individual sentences within a post as adaptive, maladaptive, or neither. This method is sensitive to sentence segmentation.
- Generation-based: Two separate LLMs are trained to directly extract sentences or phrases demonstrating adaptive and maladaptive evidence from the entire post. This approach is highlighted as more effective.
- Overall Well-being Rating: A fine-tuned LLM predicts a well-being score (1-10) for each post, using the post content and the identified adaptive/maladaptive evidence.
- Post-level Summary: This task generates a summary for an individual post, capturing the interplay between adaptive and maladaptive states and the well-being score. They utilize in-context learning (ICL) with few-shot demonstrations, selecting the most semantically similar example from the training set to guide the LLM.
- Timeline-level Summary: Similar to post-level summaries, but this task aggregates information from all posts within a user's timeline to provide a dynamic narrative of mental state fluctuations over time.
Datasets
The study utilizes data from the CLPsych 2025 shared task. This dataset provides social media timelines, annotated evidence for adaptive and maladaptive self-states, overall well-being scores for each post, and both post-level and timeline-level summaries. The task is organized around the MIND framework, which conceptualizes human experience as evolving self-states.
Key Insights and Ideas
- Generative Models Excel at Evidence Identification: The paper's key finding is that generation-based LLMs significantly outperform classification-based models in identifying adaptive and maladaptive evidence from social media posts. This suggests that directly extracting relevant text is more robust than sentence-level classification.
- Hybrid Approach for Comprehensive Modeling: Combining classification and generation, and then integrating this evidence into subsequent tasks (well-being rating and summarization), proves effective. The extracted evidence is crucial for downstream tasks.
- In-Context Learning for Summarization: For summarization tasks, leveraging in-context learning with few-shot examples, particularly by finding semantically similar posts, yields better results than fine-tuning smaller models.
- Timeline Summarization Remains a Challenge: Despite the success in evidence identification and well-being rating, the paper acknowledges that current LLMs struggle with generating accurate and insightful timeline-level summaries, even with advanced prompting techniques.
Key Results
- The generative-based evidence identification model showed a marked advantage in accuracy compared to classification-based models and baselines on Task A.1.
- For overall well-being rating (Task A.2), their proposed method (using fine-tuned Qwen2.5-7B) outperformed baselines like Llama 3.1 and BERT.
- For post-level summaries (Task B), their approach using Qwen2.5-32B and ICL achieved competitive results, with the generative evidence model showing strong performance.
- However, for timeline-level summaries (Task C), their method did not outperform baselines, indicating a limitation in current LLM capabilities for long-term temporal summarization.
Limitations and Future Work
The authors acknowledge two primary limitations:
- Limited LLM Evaluation: They only evaluated two LLMs (Qwen2.5-7B and Qwen2.5-32B) due to time constraints, and more established models like Llama 3.1-8B/70B were not included.
- Timeline Summarization Performance: Their method did not outperform baselines on timeline-level summarization.
Future work will focus on addressing these limitations, particularly by exploring methods to improve timeline-level summarization, such as incorporating post windows into the summarization process. They also emphasize the importance of ethical considerations, ensuring data privacy and responsible use of the developed methods.
Summary updated