TL;DR
This paper presents a novel approach for analyzing mental health dynamics in social media timelines using a pre-trained Large Language Model (LLM) with in-context learning. By employing a two-step sampling and selection process, the authors aim to improve the quality of generated rationales and summaries, demonstrating promising results in capturing user mental states.
What Problem Does This Paper Address?
The CLPsych 2025 shared task focuses on understanding how mental health states evolve over time within social media timelines. This involves not only identifying adaptive and maladaptive mental states but also generating human-understandable rationales that explain these changes. Existing methods often struggle to provide nuanced, dynamic insights into mental health, especially when dealing with the sheer volume and complexity of social media data. The challenge lies in extracting meaningful evidence from user posts and synthesizing it into coherent summaries that reflect the temporal progression of a user's mental well-being.
Methods
The core of the proposed method is the utilization of a pre-trained Large Language Model (LLM), specifically meta-llama/Meta-Llama-3-8B-Instruct, combined with in-context learning. To enhance the LLM's performance without extensive fine-tuning, a two-step procedure is implemented:
- Sampling: For each user post or timeline, a random subset of the provided training data is sampled to serve as context for the LLM's prompt. This is repeated multiple times to generate a diverse set of potential outputs.
- Selection: The LLM-generated outputs are then mapped into a vector space using the
sentence-transformers/all-MiniLM-L6-v2model. Gaussian kernel density estimation, with Scott's Rule for bandwidth selection, is applied to identify the most likely and highest-density output among the sampled candidates. This selection mechanism aims to filter out less coherent or less relevant generations.
For tasks A.1 and A.2 (evidence extraction and well-being scoring), a single prompting template is used, incorporating past user post analyses as in-context examples. For task C (timeline summarization), a separate template is designed, using past example timelines as context, with a limit of three such examples to avoid performance degradation.
Datasets
The paper utilizes the dataset provided for the CLPsych 2025 shared task. This dataset contains annotations for evidence aligned with the ABCD (Affect, Behavior, Cognition, and Desire) framework, well-being scores, and expert summaries at both post-level and timeline-level. The dataset is structured into four tasks: A.1 (evidence extraction), A.2 (well-being scoring), B (post summarization), and C (timeline summarization).
Key Insights and Ideas
- In-Context Learning as a Powerful Tool: The paper demonstrates that even without task-specific fine-tuning, pre-trained LLMs can be effectively guided to perform complex mental health analysis tasks through well-crafted in-context learning prompts.
- Sampling and Selection for Quality Control: The proposed two-step sampling and selection process is a crucial innovation. It addresses the inherent variability in LLM outputs by generating multiple candidates and then using a density-based approach to select the most representative one, thereby improving the reliability and quality of the generated content.
- Focus on Negative States: The results indicate that the LLM tends to focus more on extracting evidence related to maladaptive states than adaptive ones. This is attributed to the LLM's alignment with recognizing negative inputs for safeguarding purposes, a common characteristic of many public LLMs.
- Hallucination as a Persistent Challenge: Despite the promising results, the paper acknowledges that the system can still generate contradictory analyses, leading to concerns about hallucination, a known issue with LLMs.
Key Results
The paper reports that their method achieved "relatively good performance overall." Specifically:
- Task A.1 (Evidence Extraction): The system performed relatively better in evidence extraction compared to well-being scoring and summary generation. The ISM (ours) team ranked 4th in Recall for Task A.1.
- Task A.2 (Well-being Scoring): The system also showed a tendency to focus more on problematic well-being. The ISM (ours) team ranked 7th in MSE for Task A.2.
- Tasks B and C (Summarization): The system generated relatively good summaries that were highly consistent with expert annotations. The ISM (ours) team ranked 6th in Mean Consistency for Task B and 9th for Task C.
- Contradiction in Summaries: While summaries were generally good, the "max contradiction" metric indicated instances of contradictory analysis, highlighting the hallucination issue.
Limitations and Future Work
The authors acknowledge two primary limitations:
- Lack of Domain Knowledge Guarantee: The LLM, trained on general web data, may not possess the specialized professional knowledge required for a deep understanding of mental health nuances within the task's domain. Fine-tuning with high-quality professional data could mitigate this.
- Limited Domain Context Understanding: While in-context learning helps, the LLM might not fully grasp the specific context due to the absence of guaranteed domain knowledge in its pre-training.
Future work directions suggested include:
- Using Stronger LLMs: Exploring more powerful LLMs could potentially yield better performance.
- Fine-tuning with Domain Knowledge: Adapting the LLM through fine-tuning on domain-specific data is a promising avenue for improving its understanding and accuracy.
Summary updated