An Investigation of Deep Learning Systems for Suicide Risk Assessment

Michelle Morales, Prajjalita Dey, Thomas Theisen, Danny Belitz, Natalia ChernovaNAACL 2019

TL;DR

This paper explores the effectiveness of deep learning models for suicide risk assessment using Reddit data. The authors found that while deep learning approaches like CNNs and LSTMs show promise, especially with novel features like personality insights, they face challenges with imbalanced and small datasets, highlighting the need for further research in this critical area.

What Problem Does This Paper Address?

Suicide is a significant public health issue, with millions of Americans living in areas with limited mental health resources. Predicting suicide attempts has historically been difficult, with accuracy near chance. Machine learning offers a potential solution to identify individuals at higher risk, but its effectiveness, particularly with advanced deep learning techniques, needs further investigation. This paper specifically addresses the challenge of applying deep learning to suicide risk assessment using publicly available Reddit data, aiming to improve identification of at-risk individuals.

Methods

The researchers explored several deep learning architectures, including Convolutional Neural Networks (CNNs), Long Short-Term Memory networks (LSTMs), and a novel Neural Network Synthesis (NeuNetS) approach.

  • CNNs: These models were applied to text classification tasks, leveraging various word embeddings.
  • LSTMs: This approach focused on transforming a user's post history into a sequence of tone weights over time to predict risk.
  • NeuNetS: This system automates the design of deep neural networks, optimizing architecture and individual layer components. It was designed to synthesize the best deep learning model for a given dataset.

In addition to deep learning, the paper also discusses traditional machine learning approaches and various feature engineering techniques. These include:

  • Bag of Words: Using TF-IDF weighting for unigrams and bigrams.
  • Topics: Employing Latent Dirichlet Allocation (LDA) to extract topic distributions.
  • Syntax & Named Entities: Utilizing part-of-speech (POS) tags and named entity recognition (NER).
  • Word Embeddings: Exploring Skip-gram and FastText embeddings.
  • Novel Features: A key contribution is the exploration of personality features extracted via the IBM Watson Personality Insights API, including Big Five traits, Needs, and Values. Tone features from the IBM Watson Tone Analyzer were also investigated.

Datasets

The primary dataset used in this study is from the 2019 CLPsych Shared Task. This dataset consists of Reddit posts from users who have posted on the r/SuicideWatch subreddit. The posts are annotated into four categories: No Risk, Low Risk, Moderate Risk, and Severe Risk. The task involved predicting suicide risk using different subsets of user posts:

  • Task A: Focused on r/SuicideWatch posts only.
  • Task B: Included all r/SuicideWatch posts plus posts from other subreddits by the same users.
  • Task C: Focused on non-r/SuicideWatch posts from users.

The dataset includes post identifiers, user identifiers, timestamps, subreddit names, post titles, and post bodies.

Key Insights and Ideas

  • Deep Learning Shows Promise, But Data is Key: While deep learning models like CNNs and LSTMs demonstrated competitive performance, their effectiveness was significantly impacted by the size and balance of the training data. The paper suggests that deep learning might be more suitable for tasks with larger, more balanced datasets.
  • Novel Features Offer New Avenues: The integration of personality features (e.g., Big Five traits) and tone analysis provided valuable insights, suggesting that understanding a user's psychological profile can be crucial for suicide risk assessment.
  • NeuNetS Faces Data Challenges: Despite its sophisticated architecture synthesis capabilities, NeuNetS struggled with the imbalanced and relatively small datasets of the CLPsych task, leading to lower-than-expected performance. This highlights that even advanced automated design tools can be limited by data constraints.
  • Post-Level Training Outperforms User-Level: For Tasks A and B, models trained at the post level and then aggregated to the user level generally outperformed models trained directly at the user level.

Key Results

  • Task A: The NeuNetS system achieved the highest macro F1-score of .57 during the evaluation phase. However, on the test set, a CNN (Skip-gram) model achieved an accuracy of .52 and a macro F1 of .31, while NeuNetS reported a low F1-score.
  • Task B: NeuNetS also performed highest in the evaluation phase with a macro F1 of .48. On the test set, an LSTM (Tone) model achieved an accuracy of .42 and a macro F1 of .30.
  • Task C: This task, which had more data per user, saw a kNN (Big 5 + Values) model achieve an accuracy of .44 and a macro F1 of .17.
  • Overall: The paper found that systems trained at the post level generally outperformed user-based systems.

Limitations and Future Work

  • Data Imbalance and Size: The authors acknowledge that the imbalanced and limited size of the training data for Tasks A and B posed significant challenges, particularly for the NeuNetS system. This imbalance may have caused models to "steer in the wrong direction" during training.
  • NeuNetS Aggregation: The majority voting mechanism used by NeuNetS for aggregating post-level predictions to user-level did not allow for predictions of certain risk labels (b or c), potentially contributing to its low F1 scores. The authors suggest exploring alternative aggregation methods, such as averaging confidence scores.
  • Future Directions: The paper suggests that despite the challenges, there is still promise in deep learning systems for suicide risk assessment, especially for tasks facing data size limitations. Future work could focus on developing better methods for handling imbalanced data, exploring more advanced aggregation techniques, and investigating the impact of larger, more diverse datasets. The potential of novel features like personality insights also warrants further exploration.

Summary updated

BibTeX
@inproceedings{morales2019investigation,
  author = {Michelle Morales and Prajjalita Dey and Thomas Theisen and Danny Belitz and Natalia Chernova},
  title = {{An Investigation of Deep Learning Systems for Suicide Risk Assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure