TL;DR
This paper introduces the CLPsych 2019 shared task, which aimed to predict the degree of suicide risk from Reddit posts. By moving beyond binary classification to a multi-level assessment, the research provides valuable insights into the potential of social media data for mental health screening and highlights challenges in accurately identifying moderate risk levels.
What Problem Does This Paper Address?
Predicting suicide risk is notoriously difficult, even for trained clinicians. Traditional methods, like direct questioning, are often unreliable, with a high percentage of individuals who die by suicide denying suicidal thoughts when asked. Furthermore, research has shown little improvement in the predictive ability for suicidal thoughts and behaviors over the past 50 years. This is partly because these behaviors rarely occur in controlled research settings.
Social media, however, offers a unique opportunity to observe individuals' thoughts and feelings "in situ" during the "clinical whitespace" between healthcare encounters. While previous work has explored using social media for binary suicide risk screening, this paper addresses two key limitations: 1) the need for a finer-grained assessment of risk (beyond just "at risk" or "not at risk") to prioritize interventions, and 2) the difficulty in obtaining and sharing sensitive social media data for research.
Methods
The CLPsych 2019 shared task focused on predicting suicide risk from Reddit posts. The core of the methodology involved:
- Multi-level Risk Assessment: Instead of a simple binary classification, the task defined four levels of suicide risk: No Risk, Low Risk, Moderate Risk, and Severe Risk. This allows for a more nuanced understanding of an individual's potential danger.
- Data Collection and Annotation: The dataset was derived from publicly available Reddit posts, specifically focusing on users who posted in the r/SuicideWatch subreddit. A subset of these users was then annotated for risk level by both experts and crowdsourced workers. The paper notes that while expert annotations are considered more accurate, crowdsourced annotations were used for training and testing to simplify machine learning development and facilitate future comparisons.
- Three Task Variations:
- Task A: Assessed risk based solely on posts from the r/SuicideWatch subreddit.
- Task B: Expanded on Task A by including all of a user's Reddit posts, not just those from r/SuicideWatch, to see if broader context improved predictions.
- Task C: Focused on screening users for risk using only their non-SuicideWatch posts, simulating a scenario where individuals haven't explicitly signaled distress.
- De-identification: Significant steps were taken to de-identify the data, including replacing user IDs with arbitrary numeric identifiers and using special tokens to mask personally identifiable information like IP addresses, emails, URLs, and person entities. Posts containing Arabic were also filtered out.
Datasets
The primary dataset used for this shared task was introduced by Shing et al. (2018). It comprised:
- Candidate Users: 11,129 users who had posted to the r/SuicideWatch subreddit.
- Posts: A total of 1,556,194 posts from these users across Reddit.
- Control Users: A comparable number of users who did not post on r/SuicideWatch.
A randomly selected subset of 621 users was annotated for risk level. The data was split into training (80%) and testing (20%) sets, with proportional splitting to maintain label distribution.
Key Insights and Ideas
- Beyond Binary: The move to a multi-level risk assessment is crucial for practical intervention prioritization, as a large number of "at-risk" flags from a binary system would overwhelm clinical capacity.
- Context Matters (Sometimes): Comparing Tasks A and B suggests that including a user's broader Reddit activity (beyond r/SuicideWatch) generally did not introduce harmful noise and could even improve performance, particularly for identifying severe risk.
- Moderate Risk is Tricky: Systems, and even human annotators, struggled to accurately distinguish between low, moderate, and no risk. This is partly attributed to the smaller training data for these intermediate categories.
- Distinct Vocabulary for Severe Risk: Users exhibiting severe risk appear to use a distinct vocabulary, suggesting the potential benefit of building separate language models for SuicideWatch and non-SuicideWatch content.
- Over-prediction of Severe Risk: Systems tended to over-predict severe risk when the true label was moderate risk, highlighting the difficulty in this distinction and the value of metrics like "urgent F1" which capture this urgency.
- False Positives and Relatives: A notable source of false positives involved users seeking advice for a relative or friend, indicating a need for models that can differentiate these scenarios.
Key Results
- Task A (SuicideWatch posts only): The CLaC team achieved the best official macro-F1 score of 0.481 with a hybrid SVM and deep neural network approach. However, an alternative variation of their approach yielded the best unofficial score of 0.533.
- Task B (All Reddit posts): The HLAB team achieved the best official macro-F1 score of 0.457. Their best unofficial score was 0.504, using BERT features.
- Task C (Non-SuicideWatch posts only): The CAMH system achieved the best unofficial F1 score of 0.278. The CLaC team's primary system achieved an official score of 0.268.
- General Performance: Across tasks, systems performed better at identifying the extreme risk labels (No Risk and Severe Risk) compared to the intermediate labels (Low and Moderate Risk).
Limitations and Future Work
- Lack of Outcome Data: The study acknowledges the limitation of not having actual suicide attempt or completion data, relying instead on human annotations of risk level.
- Crowdsourced Annotations: While practical, using crowdsourced annotations as ground truth may not be as accurate as expert judgments.
- Data Imbalance: The uneven distribution of labels, particularly for low and moderate risk, presented a challenge for model training.
- Future Directions: The authors suggest building separate language models for SuicideWatch and non-SuicideWatch content, developing models to distinguish between self-risk and concern for others, and creating more robust infrastructure for handling sensitive mental health data with full consent. They also aim to facilitate community-level evaluation against expert judgments and explore the use of consented, donated data for future research.
Summary updated