CLaC at CLPsych 2019: Fusion of Neural Features and Predicted Class Probabilities for Suicide Risk Assessment Based on Online Posts

Elham Mohammadi, Hessam Amini, Leila KosseimNAACL 2019

TL;DR

This paper presents a winning approach to the CLPsych 2019 shared task for suicide risk assessment from Reddit posts. Their method fuses features from multiple neural networks with predicted class probabilities, feeding them into an SVM classifier, and achieved first place in two out of three tasks. This highlights the power of ensemble methods and combining diverse feature representations for sensitive clinical NLP tasks.

What Problem Does This Paper Address?

The paper tackles the critical challenge of automatically assessing suicide risk based on online user-generated content, specifically Reddit posts. This is a complex task because suicide risk is a nuanced, multi-faceted issue, and online text can be highly variable and indirect. The CLPsych 2019 shared task aimed to classify users into four risk categories: no risk, low risk, moderate risk, and severe risk. The task was further divided into three sub-tasks: predicting risk based solely on the r/SuicideWatch subreddit, considering all subreddits, and considering all subreddits except r/SuicideWatch (for screening purposes). Accurately identifying individuals at risk is crucial for timely intervention and support.

Methods

The core of the CLaC team's approach is an ensemble method that combines the strengths of multiple neural network sub-models. Here's a breakdown:

  • Neural Sub-models: They employed eight distinct neural network sub-models. Each sub-model used one of two pre-trained word embedding types: GloVe or ELMo. For the hidden layers, they experimented with four architectures: Convolutional Neural Networks (CNNs), Bidirectional Recurrent Neural Networks (Bi-RNNs), Bidirectional Long Short-Term Memory networks (Bi-LSTMs), and Bidirectional Gated Recurrent Units (Bi-GRUs).
  • Pooling: To create a fixed-size representation from the variable-length outputs of the hidden layers, they used three pooling strategies: Average (AVG), Max, and Attention (ATTN).
  • Attention Mechanism: A key component is an attention mechanism applied at two levels. First, within each sub-model, an attention mechanism (similar to Bahdanau et al., 2014) was used to weigh the importance of different parts of the post's representation. Second, a user-level attention mechanism was employed to assign weights to individual posts from a user, hypothesizing that not all posts contribute equally to signaling mental state. This user-level attention calculates a weighted average of post representations.
  • Fusion Component: The outputs from the eight sub-models were combined. This fusion component first concatenated the neural features (user representations) from each sub-model. Then, it selected a subset of these features based on their mutual information with the final classes. Crucially, it also incorporated the predicted class probabilities from the softmax output of each sub-model.
  • Final Classifier: The fused features (a combination of selected neural features and predicted probabilities) were fed into a Support Vector Machine (SVM) classifier for the final prediction.

Datasets

The primary dataset used was from the CLPsych 2019 shared task, which consisted of Reddit posts from users. The task provided labeled data for training and evaluation. The paper mentions that 33% of the training data was set aside for validation during the development of the sub-models and hyperparameter tuning.

Key Insights and Ideas

  • Ensemble Power: Combining multiple diverse neural models (different embeddings, architectures, and pooling) significantly boosts performance. This suggests that no single model architecture is universally optimal for this task.
  • Fusion of Features and Probabilities: The success of "Run 3" (which used both neural features and predicted probabilities) across tasks A and C indicates that these two types of information are complementary and valuable. Neural features capture deeper semantic and contextual information, while predicted probabilities offer a direct signal from each sub-model.
  • Attention is Crucial: The use of attention mechanisms at both the token/post level (within sub-models) and the user level (for aggregating posts) is highlighted as important. This allows the model to focus on the most relevant parts of the text and the most indicative posts.
  • Distinguishing Risk Levels: The paper notes that using only neural features (Run 1) was best for distinguishing between "no-risk" users and those requiring attention (flagged F1 score), suggesting neural features excel at capturing subtle indicators of distress.

Key Results

The CLaC team achieved remarkable success, ranking first in two out of the three tasks (Task A and Task C) of the CLPsych 2019 shared task.

  • Task A & C: Run 3, which combined neural features and predicted probabilities, yielded the highest macro-averaged F1 scores.
  • Task B: Run 2, which used only predicted probabilities, achieved the highest F1 score.
  • Flagged F1: Across all tasks, Run 1 (neural features only) performed best for the "flagged" metric, indicating its strength in identifying users who need attention.

The paper also mentions that the average size of posts across all tasks was approximately 78 tokens, and for sub-model processing, they used either the first 200 or 400 tokens, depending on which yielded better validation performance.

Limitations and Future Work

  • Computational Resources: The authors mention that limitations in computational resources led to using mini-batches with a maximum size of 32 at the post level for each user. This could potentially be a bottleneck for larger datasets or more complex models.
  • Hyperparameter Tuning: While they used validation data for hyperparameter tuning, the sheer number of sub-models and hyperparameters suggests that exhaustive tuning might be challenging.
  • Future Work: The paper implicitly suggests that further exploration of ensemble strategies, attention mechanisms, and feature fusion techniques could lead to even better performance. The success of combining diverse information sources points towards continued research in this direction for clinical NLP tasks.

Summary updated

BibTeX
@inproceedings{mohammadi2019clac,
  author = {Elham Mohammadi and Hessam Amini and Leila Kosseim},
  title = {{CLaC at CLPsych 2019: Fusion of Neural Features and Predicted Class Probabilities for Suicide Risk Assessment Based on Online Posts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure