TL;DR
This paper presents a machine learning approach using decision trees and a rich set of linguistic features (dictionaries) to predict suicide risk from Reddit posts. The study highlights the predictive power of social and moral language categories, as well as individual word correlates, offering insights into the linguistic markers of mental distress.
What Problem Does This Paper Address?
Predicting suicide risk from online text is a critical challenge with significant implications for mental health support. Researchers are exploring how language used on social media platforms like Reddit can serve as an indicator of psychological distress, including suicidal ideation. This paper addresses the CLPsych 2019 Shared Task, which aimed to predict suicide risk levels (none, low, moderate, severe) based on user posts from the r/SuicideWatch subreddit. The challenge lies in identifying reliable linguistic features that can accurately capture these nuanced risk levels, especially given the inherent noise and variability in user-generated content.
Methods
The authors employed a recursive partitioning algorithm, specifically decision trees, to build their predictive models. The core of their approach involved a comprehensive feature engineering process. They utilized a wide array of pre-existing linguistic dictionaries and lexicons (listed in Table 1), covering aspects like sentiment, emotion, social interaction, and moral foundations. In addition to dictionary scores, they also incorporated counts of individual words.
A significant part of their methodology was dedicated to preprocessing and spelling correction. This involved cleaning the text by standardizing encoding, removing code elements, and handling common typing variations. They developed a two-phase spelling correction process using the hunspell package and measuring edit distances to map misspelled words to their correct forms. This meticulous cleaning aimed to ensure the accuracy of the linguistic features extracted.
For model training, they used the rpart package for decision trees. They trained models for each task (A, B, and C, focusing on different subsets of data) and then aggregated predictions at the user level to assign a final risk rating. They also explored other models like linear regressions, logistic regressions, multinomial logistic regressions, and random forests but found decision trees to be most effective for their approach.
Datasets
The primary dataset used in this study was derived from Reddit posts, specifically from the r/SuicideWatch subreddit. The data was pre-labeled by clinical psychologists with suicide expertise into four risk categories: none, low, moderate, and severe. The paper mentions tasks A, B, and C, which likely represent different subsets or configurations of this Reddit data for the shared task. The authors filtered out posts not belonging to the task A or B sets and those with only "nan" as the body, resulting in 11,856 posts from 329 users for their analysis.
Key Insights and Ideas
- Social and Moral Language are Key: The study found that categories related to social and moral language were particularly important for predicting suicide risk. This suggests that how individuals discuss relationships, ethics, and societal norms can be strong indicators of their mental state.
- Self-Focus and Negative Affect: Consistent with prior research, the use of first-person singular pronouns (self-focus) and negative emotion words were expected and found to be correlated with higher suicide risk. This reinforces the idea that internalizing and negative self-talk are significant markers.
- Fine-Grained Data-Driven Approaches: The paper emphasizes the value of data-driven approaches that go beyond broad categories. Word-level correlates of risk levels revealed both expected and novel associations, suggesting that detailed linguistic analysis can uncover subtle but important patterns.
- Simpler Punctuation as a Signal: Interestingly, the use of more periods and fewer parentheses in severe-risk posts was observed. The authors hypothesize this might indicate simpler, more direct writing, potentially reflecting a more urgent or certain desire to die.
- Intuition vs. Declarative Knowledge: The authors propose that tasks like this can help bridge the gap between clinicians' intuitive assessments of risk and the explicit linguistic features that drive those intuitions, potentially improving both predictive models and clinical practice.
Key Results
The models achieved moderate performance on the official test sample, with F1 scores varying across tasks (e.g., Task A achieved an F1 score of .667 for the 'a' rating, .200 for 'b', .140 for 'c', and .600 for 'd'). The decision trees tended to be simple, often predicting extreme ratings (high risk) due to the distribution of ratings in the dataset.
Qualitatively, higher-risk posts were characterized by:
- Higher "Clout" scores (more use of "I") and lower rates of discussing family.
- More references to care, vice, and virtue.
- More periods, fewer parentheses, and more hyperbolic statements.
- Fewer third-person singular pronouns.
- More words indicating certainty, finality, or black-and-white thinking (e.g., "every," "anymore").
- More focus on physical harm and life/death.
- Increased use of negations.
- Swearing was also indicative of higher risk.
- Self-focused pronouns ("I," "me," "myself") were strong correlates of the highest risk level.
Lower-risk posts, conversely, tended to be more social, using more communicative words and words connoting warmth, along with more "we" and specific family references.
Limitations and Future Work
The authors acknowledge that their models tended to capitalize on prior rating distributions, leading to predictions of extreme ratings. This is exacerbated by the fact that some users had multiple posts, allowing idiosyncrasies to influence the overall user rating.
A key limitation is that the expert coders had no more information than the models, meaning there's a possibility of false positives in every risk category. The authors also note that the study doesn't know the actual outcomes for the users (e.g., if they died by suicide), which limits the definitive interpretation of the risk ratings.
Future work could explore more sophisticated models and feature sets. The authors suggest that determining the behavioral variables influencing clinicians' intuitions about risk is a promising direction, especially in the context of noisy behavioral samples from online platforms. They also mention that mixed-effects models were considered but did not work well for Task A due to users having very few posts.
Summary updated