Automatic identification of suicide notes with a transformer-based deep learning model

Tianlin Zhang, Sophia AnaniadouOpenReview 2021 · OpenReview

TL;DR

This paper introduces TransformerRNN, a novel deep learning model that effectively identifies suicide notes from online text by combining Transformer encoders and BiLSTMs. The model achieves state-of-the-art performance, outperforming existing methods and offering a promising tool for developing suicide prevention technologies on social media.

What Problem Does This Paper Address?

Suicide is a critical global health issue, and with the rise of social media, individuals are increasingly posting messages that may indicate suicidal intent or serve as suicide notes. Automatically identifying these notes is crucial for understanding mental health trends and developing timely interventions. However, existing methods struggle to capture both the nuanced local details and the broader semantic context within these texts. This paper aims to bridge this gap by proposing a deep learning model capable of extracting these complex linguistic features for accurate suicide note detection.

Methods

The core of this research is the TransformerRNN model. It's a hybrid architecture designed to leverage the strengths of both Transformer encoders and Bi-directional Long Short-Term Memory (BiLSTM) networks.

Here's a breakdown of its components:

  • Input Embeddings: The model begins by converting input text (words) into dense vector representations using pre-trained GloVe embeddings.
  • Transformer Encoder: This component is key to capturing contextual and long-term dependencies. It uses a multi-head self-attention mechanism to weigh the importance of different words in relation to each other, regardless of their distance in the sequence. Positional encodings are added to retain word order information.
  • BiLSTM Layer: The output from the Transformer encoder is then fed into a BiLSTM. This recurrent neural network processes the sequence in both forward and backward directions, further enhancing the model's ability to understand sequential and contextual information.
  • Max-Pooling Layer: This layer helps to extract the most salient features from the BiLSTM's output, summarizing the important semantic information.
  • Classification Layer: Finally, a softmax layer predicts the probability of the input text belonging to one of the three classes: suicide notes, last statements, or neutral posts.

The novelty lies in the synergistic combination of the Transformer's ability to model global dependencies with the BiLSTM's capacity for sequential understanding, creating a powerful tool for nuanced text analysis.

Datasets

The study utilizes a custom-built dataset for training and evaluation, comprising three classes:

  • Suicide Notes: 659 samples, collected from existing corpora and Kaggle. The authors note that while some data is confirmed from individuals who died by suicide, other data from Kaggle consists of posts from users expressing suicidal thoughts, with a statistical analysis confirming linguistic similarities.
  • Last Statements: 431 records from prisoners who received the death penalty in Texas between 1982 and 2017, provided by the Department of Criminal Justice.
  • Neutral Posts: 2000 samples collected from ten different subreddits (e.g., r/fitness, r/parenting) that do not contain suicidal content.

The data was anonymized, and all labels were manually checked for accuracy.

Key Insights and Ideas

  • Social Media as a Suicide Note Repository: The paper highlights a concerning trend of individuals using social media platforms to post suicide notes, underscoring the need for automated detection tools.
  • Limitations of Traditional Methods: Conventional machine learning approaches, while useful, often rely on handcrafted features and expert knowledge, which can be labor-intensive and may not fully capture the complex linguistic nuances of suicide notes.
  • Power of Deep Learning for Context: Deep learning models, particularly those incorporating attention mechanisms and recurrent structures, are shown to be significantly more effective at extracting the rich semantic and contextual information present in these sensitive texts.
  • Transformer-BiLSTM Synergy: The combination of Transformer encoders for global context and BiLSTMs for sequential understanding proves to be a potent strategy for this classification task.
  • Linguistic Differences in Distress: The dataset analysis reveals distinct linguistic patterns across the three categories, with suicide notes often exhibiting lower "clout" (social confidence) and a focus on personal emotions, while last statements might show more resignation or repentance.

Key Results

The TransformerRNN model achieved impressive results on the test set:

  • Overall Performance: The model achieved a weighted average Precision (P) of 95.0%, Recall (R) of 94.9%, and F1-score of 94.9%.
  • Outperforming Baselines: TransformerRNN significantly outperformed traditional machine learning models (J48, Naive Bayes, Bayes Net, LMT) and other deep learning models, including CNN, BiLSTM, BiLSTMAttention, and DLSTMAttention.
  • Suicide Note Specialization: While performing well across all classes, the model showed particular strength in identifying suicide notes, which is the primary focus of the research.
  • Ablation Studies: Removing components like the max-pooling layer or the BiLSTM layer led to a decrease in performance, confirming the effectiveness of each part of the TransformerRNN architecture.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Dataset Size: The dataset of suicide notes, while augmented, is still relatively small (659 samples). Future work could focus on collecting more precise data from diverse online sources.
  • Data Certainty: While efforts were made to ensure data quality, some "suicide notes" were identified from users expressing suicidal thoughts rather than confirmed cases of suicide. Future research could explore more robust methods for data validation.
  • Interpretability: Deep learning models, including TransformerRNN, are often considered "black boxes." While effective, their lack of direct interpretability can be a challenge for clinical applications. Future work might explore methods to enhance the interpretability of these models.
  • Alternative Approaches: The paper suggests that semi-supervised and unsupervised learning approaches could also be valuable for suicide note identification.

Summary updated

Abstract

Suicide is one of the leading causes of death worldwide. At the same time, the widespread use of social media has led to an increase in people posting their suicide notes online. Therefore, designing a learning model that can aid the detection of suicide notes online is of great importance. However, current methods cannot capture both local and global semantic features. In this paper, we propose a transformer-based model named TransformerRNN, which can effectively extract contextual and long-term dependency information by using a transformer encoder and a Bi-directional Long Short-Term Memory (BiLSTM) structure. We evaluate our model with baseline approaches on a dataset collected from online sources (including 659 suicide notes, 431 last statements, and 2000 neutral posts). Our proposed TransformerRNN achieves 95.0%, 94.9% and 94.9% performance in P, R and F1-score metrics respectively and therefore outperforms comparable machine learning and state-of-the-art deep learning models. The proposed model is effective for classifying suicide notes, which in turn, may help to develop suicide prevention technologies for social media.
BibTeX
@misc{zhang2021automatic,
  author = {Tianlin Zhang and Sophia Ananiadou},
  title = {{Automatic identification of suicide notes with a transformer-based deep learning model}},
  year = {2021},
  url = {https://openreview.net/forum?id=GL7By31CYDV},
  howpublished = {OpenReview},
}
Figure