Determining a Person’s Suicide Risk by Voting on the Short-Term History of Tweets for the CLPsych 2021 Shared Task

Ulya Bayram, Lamia BenhibaNAACL 2021

TL;DR

This paper explores using tweet history to predict suicide risk, finding that a short-term (30-day) window is more effective than a longer one. They propose a tweet-level classification and majority voting approach, achieving high accuracy for immediate risk detection, and suggest temporal decay as a future direction for longer-term prediction.

What Problem Does This Paper Address?

Suicide is a major public health concern, and the increasing use of social media presents an opportunity for early detection. Identifying individuals at risk of suicide based on their online activity, particularly their tweets, is a challenging but crucial task. This paper addresses the need for effective tools to detect suicidal ideation from Twitter data, focusing on the temporal aspect of this risk and how it manifests in short-term versus long-term tweet histories. The challenge lies in the noisy nature of social media data and the scarcity of labeled examples, making it a small-dataset classification problem.

Methods

The authors adopted a tweet-level classification approach, inspired by real-world auto-detection systems that monitor posts in real-time. After cleaning tweets (removing mentions, URLs, punctuation, etc.), they experimented with both traditional Machine Learning (ML) and Deep Learning (DL) methods.

For ML, they focused on n-gram features and employed feature selection to identify the most informative ones. They evaluated Logistic Regression (LR), Naive Bayes (GNB, MNB), and Linear Support Vector Machines (1SVM). A key technique was majority voting (MV), where individual tweet risk scores were aggregated to determine an overall user risk score. They also explored an ensemble classifier (wEns) that weighted the predictions of multiple ML models based on their performance.

For DL, they utilized a pre-trained transformer model (BERT) followed by a multi-layer bi-directional GRU architecture (GRU-Bert) to capture sequential text patterns. Similar to the ML approach, tweet-level predictions were combined using majority voting for user-level risk assessment.

Datasets

The study used labeled datasets from the CLPsych 2021 Shared Task. These datasets contained tweets from users who had attempted suicide and control users.

  • Subtask 1: Tweets collected from 30 days before the adverse event. The labeled collection consisted of 57 suicidal users and 57 control users.
  • Subtask 2: Tweets collected from six months before the adverse event. The labeled collection consisted of 82 suicidal users and 82 control users.

The authors note that these are small datasets, posing a challenge for classification.

Key Insights and Ideas

  • Short-term signals are stronger: The most significant finding is that tweet data from the 30 days leading up to a suicidal event is much more effective for detecting immediate risk than data from a six-month period. This suggests that suicidal ideation signals become more pronounced and relevant closer to the event.
  • Majority voting works well for short-term risk: Aggregating tweet-level risk scores through majority voting proved to be a viable and effective strategy for short-term suicide risk prediction.
  • Long-term data introduces noise: The performance drop on the six-month dataset indicates that older tweets, especially those lacking clear suicidal ideation, can act as noise, weakening the overall signal for long-term risk.
  • Simple ML can be powerful: For this specific task and dataset size, well-tuned ML methods with feature selection, especially when combined in an ensemble, outperformed more complex DL models.
  • Temporal decay is promising: The authors hypothesize that incorporating a time-decay mechanism into the voting scheme could significantly improve performance on longer-term datasets, giving more weight to recent tweets.

Key Results

  • Subtask 1 (30 days): The weighted ensemble (wEns) method achieved the best within-dataset performance, with an AUC of 93.2%. On the test set, wEns also provided the best F1, F2, and TPR scores, demonstrating strong generalizability for short-term risk.
  • Subtask 2 (6 months): Performance on the six-month dataset was less satisfactory. While wEns performed comparably to the baseline in terms of TPR, its other scores were lower. This highlights the difficulty in capturing long-term suicidal ideation signals from tweets.
  • DL vs. ML: Shallow DL methods (GRU-Bert) performed poorly on both within-dataset and test evaluations, often requiring significant computational resources and showing sensitivity to noise and small datasets. Simple ML methods, particularly the ensemble, delivered superior results.

Limitations and Future Work

  • Limitations:
    • The performance on the six-month dataset (Subtask 2) was unsatisfactory, indicating that current models struggle with detecting long-term suicidal ideation signals due to noise from older, less relevant tweets.
  • Shallow DL methods, despite their complexity, did not yield better results than simpler ML approaches, possibly due to overfitting or noise sensitivity on small datasets.
  • The study focused on identifying "state-level" immediate risk, rather than "trait-level" general risk.
  • Future Work:
    • Incorporate temporal aspects: The authors strongly suggest integrating temporal decay mechanisms and explicitly using tweet dates as features to improve models, especially for longer time spans.
  • Develop noise-immune models: Future research should focus on models that are less affected by irrelevant or noisy tweets.
  • Explore more complex DL architectures: Investigating deeper and more sophisticated DL models, such as those incorporating CNNs, deeper LSTMs/GRUs, and advanced word embedding techniques, is recommended.
  • Manual inspection insights: The authors note that manual inspection of tweets revealed noisy and ambiguous language, suggesting that future models might need to better handle such complexities.

Summary updated

BibTeX
@inproceedings{bayram2021determining,
  author = {Ulya Bayram and Lamia Benhiba},
  title = {{Determining a Person’s Suicide Risk by Voting on the Short-Term History of Tweets for the CLPsych 2021 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure