TL;DR
This paper introduces the CLPsych 2022 Shared Task, which focuses on automatically identifying "Moments of Change" in longitudinal social media posts to better understand mental health shifts. The task's two sub-challenges involve detecting mood changes and assessing suicide risk, aiming to bridge the gap between static individual classifications and dynamic, real-time mental health monitoring.
What Problem Does This Paper Address?
The clinical community is increasingly looking for better ways to monitor mental health conditions. While Natural Language Processing (NLP) has shown promise in identifying signals for certain conditions in text, most research has focused on classifying individuals at a single point in time (e.g., suicide risk, presence of a mental health condition). This approach misses the crucial longitudinal aspect of tracking an individual's mood and well-being over time. This paper addresses this gap by proposing a shared task that models changes in an individual's mood and their connection to suicide risk, using their online social media activity.
Methods
The CLPsych 2022 Shared Task was divided into two subtasks:
- Task A: Moments of Change (MoC) Detection: This task involved identifying specific posts or sequences of posts within a user's timeline that indicate a significant shift in mood. Two types of changes were defined: "Switch" (sudden mood shifts) and "Escalation" (gradual mood progression). This was framed as a post-level sequential classification task.
- Task B: Suicide Risk Assessment: This task, a continuation of previous work, aimed to classify an individual's suicide risk level (no/low, moderate, severe) based on their posts. This was a user-level classification task.
Participants were encouraged to explore how identifying mood changes in Task A could inform suicide risk assessment in Task B. The evaluation metrics included both post-level and timeline-level metrics for Task A, and standard classification metrics for Task B. A secure data enclave environment (NORC Data Enclave) was used to protect sensitive user data.
Datasets
The shared task utilized three datasets:
- Reddit-UMD: This dataset, derived from the r/SuicideWatch subreddit, was annotated for MoC and relabeled for suicide risk. It consists of 38K posts from 245 Reddit users.
- Reddit-New: A newly collected dataset from various mental health-related subreddits, containing posts from 83K users.
- eRisk++: This dataset, obtained through a data use agreement, contains posts and comments from users with and without self-harm conditions. The focus was on 41 users with relevant data.
Key Insights and Ideas
- The Importance of Longitudinal Modeling: The best-performing systems for Task A demonstrated that incorporating a longitudinal component into models, rather than classifying each post in isolation, is crucial for accurately capturing mood changes.
- Bridging Mood Changes and Suicide Risk: While not always leading to performance gains, the best teams for Task B often leveraged information from Task A, either by focusing on posts labeled as MoC or by jointly learning both tasks. This highlights the potential for connecting dynamic mood tracking with static risk assessment.
- Addressing Class Imbalance: Many teams tackled the inherent class imbalance in mental health data by using techniques like instance balancing, weighted loss functions, or down-sampling.
- Secure Environments are Essential: The use of a secure data enclave (NORC Data Enclave) was critical for handling sensitive mental health data, though it also presented technical challenges for participants.
Key Results
- Task A: The best-performing systems achieved high scores on both post-level and timeline-level metrics. Systems that incorporated longitudinal components, such as multi-task learning or emotionally-informed seq2seq models, performed particularly well.
- Task B: The ranking of teams for Task B varied depending on whether micro- or macro-averaged metrics were considered, largely due to the low number of users in the 'low' risk class. Teams that leveraged Task A predictions, particularly those focusing on MoC, tended to perform better.
Limitations and Future Work
- Data Size and Domain Specificity: The paper acknowledges that the dataset size can be a limitation for drawing conclusive outcomes, especially in this sensitive domain.
- Annotation Clarity: Some participants found the annotation guidelines for Task A to be unclear, suggesting a need for more examples in future longitudinal tasks.
- Secure Environment Challenges: The use of the secure data enclave, while necessary, led to technical issues for some teams, highlighting the need for robust planning and ongoing collaboration with such environments.
- Future Work: The authors emphasize the need for continued research on longitudinal tracking and modeling of user mental health within common experimental settings and secure environments. They also suggest improving communication channels and providing clearer guidelines for future shared tasks.
Summary updated