ConvSent at CLPsych 2019 Task A: Using Post-level Sentiment Features for Suicide Risk Prediction on Reddit

Kristen Allen, Shrey Bagroy, Alex Davis, Tamar KrishnamurtiNAACL 2019

TL;DR

This paper tackles the critical challenge of predicting suicide risk from social media posts, specifically on Reddit's r/SuicideWatch forum. By combining a convolutional neural network (CNN) with linguistic features from the LIWC dictionary, the authors achieve a macro-averaged F1 score of 0.50 on a challenging suicide risk prediction task, demonstrating the value of incorporating rich linguistic information into deep learning models for mental health analysis.

What Problem Does This Paper Address?

Identifying individuals at risk of suicide is a significant public health challenge. Traditional methods often rely on direct disclosure, which can be hindered by stigma and secrecy surrounding mental health issues. This paper addresses the need for supplementary methods that can infer suicide risk from publicly available data, such as social media. Specifically, it focuses on predicting suicide risk levels from Reddit posts within the r/SuicideWatch subreddit, a community where users often discuss their struggles. The challenge lies in accurately classifying users into distinct risk categories (no, low, moderate, severe) based on their textual content, a task that requires nuanced understanding of language and sentiment.

Methods

The core of the proposed approach is a convolutional neural network (CNN) architecture. The authors leverage pre-trained GloVe word embeddings to represent words within posts. For each user, posts are concatenated chronologically, forming a sequence of word embeddings. This sequence is then fed into a CNN.

A key innovation is the integration of Linguistic Inquiry and Word Count (LIWC) features. LIWC categorizes text based on psychological and linguistic dimensions (e.g., affect, cognitive processes, personal pronouns). These LIWC features are incorporated at the post level. The authors explore different configurations:

  • Affect-only model: Uses only LIWC features related to negative affect (anger, anxiety, sadness).
  • Primary model: Incorporates all 45 LIWC categories.
  • Balanced classes model: Uses custom weights to address class imbalance in the training data.
  • Leave none out model: Utilizes the entire training dataset.

The CNN architecture is designed to process these features, with specific choices for filter size, stride, and fully connected layers to capture relationships between linguistic features and predict risk.

Datasets

The primary dataset used is the University of Maryland Reddit Suicidality Dataset, made available for the CLPsych 2019 Shared Task A. This dataset includes approximately 1000 users annotated with a four-level suicide risk scale (no, low, moderate, severe) and a larger set of 20,000 unannotated users.

Key Insights and Ideas

  • LIWC features are powerful predictors: The study strongly suggests that linguistic features, particularly those captured by LIWC, are highly informative for suicide risk prediction, even outperforming models that solely rely on word embeddings.
  • Holistic user and post consideration is crucial: The CNN architecture, by processing sequences of posts and incorporating LIWC features, allows for a more holistic view of a user's mental state over time, rather than just focusing on individual posts.
  • Overfitting is a significant challenge: With a relatively small annotated dataset, models are prone to overfitting, performing well on training data but poorly on unseen data. This highlights the importance of careful feature selection and model regularization.
  • Class imbalance requires attention: The intermediate risk categories ('low' and 'moderate') are underrepresented, making them harder to predict. Techniques like custom weighting can help mitigate this.
  • Filter visualization reveals linguistic patterns: Visualizing the CNN filters reveals that individual filters learn to detect specific linguistic patterns or "character traits" associated with different risk levels, offering interpretable insights into the model's decision-making.

Key Results

The best-performing model, the "Leave none out" model, achieved a macro-averaged F1 score of 0.50 on the withheld test set. This is competitive with the best-scoring system in the task (F1=0.53). The model also demonstrated strong performance on binary splits: a macro-averaged F1 of 0.90 for the 'flagged' (some risk vs. no risk) task and 0.82 for the 'urgent' (severe risk vs. non-severe risk) task. The "Affect-only" model achieved a macro-averaged CV F1-score of 0.49, and the "Primary" model achieved a test set macro-averaged F1 of 0.37.

Limitations and Future Work

Limitations:

  • Dataset size: The authors acknowledge that the size of the annotated dataset is insufficient for training very large models without overfitting.
  • Overfitting: Despite efforts, the models still exhibit some degree of overfitting, particularly when trained for extended periods.
  • Difficulty with intermediate classes: The "Affect-only" model struggled with 'low' and 'moderate' risk categories, suggesting that a broader set of LIWC features is necessary.
  • Misclassification of severe risk: The "Leave none out" model miscategorized 42% of 'severe risk' cases as 'moderate risk', indicating room for improvement in distinguishing the highest risk levels.

Future Work:

  • Leveraging more data: The authors suggest that a larger training dataset would significantly improve performance, especially for underrepresented classes.
  • Exploring more sophisticated architectures: While CNNs are effective, exploring other deep learning architectures could yield further gains.
  • Investigating feature interactions: Deeper analysis of how different LIWC features interact could lead to more targeted feature engineering.
  • Improving interpretability: Further work on understanding what specific linguistic cues trigger high-risk predictions could enhance trust and utility.

Summary updated

BibTeX
@inproceedings{allen2019convsent,
  author = {Kristen Allen and Shrey Bagroy and Alex Davis and Tamar Krishnamurti},
  title = {{ConvSent at CLPsych 2019 Task A: Using Post-level Sentiment Features for Suicide Risk Prediction on Reddit}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure