Team 9: A Comparison of Simple vs. Complex Models for Suicide Risk Assessment

Michelle Morales, Prajjalita Dey, Kriti KohliNAACL 2021

TL;DR

This paper explores whether simpler machine learning models can outperform complex deep learning models for suicide risk assessment using social media data. The authors found that traditional models, like those using Term Frequency and Logistic Regression, performed surprisingly well, even outperforming fine-tuned BERTweet models on certain metrics, especially when dealing with limited data.

What Problem Does This Paper Address?

Suicide is a major global health concern, and accurately identifying individuals at risk is crucial. While Natural Language Processing (NLP) and Machine Learning (ML) have shown promise in predicting suicide risk from text, achieving high accuracy remains challenging. This is partly due to the difficulty in obtaining high-quality, labeled datasets, especially from sensitive sources like social media. This paper specifically addresses the CLPsych 2021 Shared Task, which aimed to predict suicide attempts within 30 days (Subtask 1) and 6 months (Subtask 2) prior, using a secure dataset of Twitter users' posts. The core question is whether sophisticated deep learning models, like BERTweet, offer a significant advantage over simpler, more traditional ML approaches in this specific, data-constrained context.

Methods

The researchers experimented with a range of models, categorizing them into three main groups:

  • Baseline Model: A standard Term Frequency (TF) model combined with Logistic Regression. This involved basic text cleaning like removing hashtags, stopwords, and tokenizing.
  • Machine Learning Models:
    • Gradient Boosting with Syntax Features: This model used manually crafted features focusing on grammatical patterns, pronoun usage, and sentence/word length.
  • Gradient Boosting with Character TF-IDF: This model used the same Gradient Boosting architecture but employed character-level TF-IDF vectors as features.
  • Ensemble Voting Classifier: This approach combined the predictions of three different classifiers (Logistic Regression, Multinomial Naive Bayes, and Random Forest) trained on n-gram TF features. A soft voting mechanism averaged their predicted class probabilities.
  • Deep Learning Models:
    • BERTweet: A pre-trained language model specifically designed for English tweets. The researchers explored two fine-tuning methods:
  • Method 1: Adding a single classification layer on top of the pre-trained BERTweet encoder and training both end-to-end.
  • Method 2: Freezing the BERTweet layers and only training an attached dense neural network.

All models, except for the deep learning ones, leveraged the same preprocessing pipeline.

Datasets

The primary dataset used was provided by the 2021 CLPsych Workshop organizers. This dataset consisted of Twitter users' posts over specific timeframes: 30 days for Subtask 1 and 182 days for Subtask 2. Crucially, it included true binary labels for past suicide attempts, a significant feature for this type of research. The data was accessed within a secure computing environment.

Key Insights and Ideas

  • Simplicity Can Be Powerful: The paper's central finding is that simpler, traditional ML models can be highly competitive, and sometimes superior, to complex deep learning models for suicide risk assessment, especially with limited data.
  • Deep Learning's Overfitting Challenge: The sophisticated BERTweet model, despite its power, struggled with overfitting on this relatively small dataset. This led to high false positive rates (FPR) and lower Area Under the Curve (AUC) scores, indicating it was too eager to predict a positive outcome.
  • Ensemble Methods Shine: When deep learning couldn't be fully applied (due to sequence length limitations in Subtask 2), ensemble methods, like the voting classifier, demonstrated strong performance, outperforming the baseline and individual gradient boosting models.
  • Data Sensitivity and Constraints: The inherent sensitivity of suicide risk data and the limitations imposed by shared task formats (like sequence length) significantly impact model choices and performance.
  • Overprediction vs. Underprediction: The authors argue that for suicide risk assessment, overpredicting risk is preferable to underpredicting, given the severe consequences of missing a potential case. This perspective influences how metrics like FPR and AUC are interpreted.

Key Results

  • Subtask 1 (30 days): While BERTweet achieved high F1, F2, and TPR, it also had a significantly higher FPR (0.818) and a lower AUC (0.413) compared to the Task Baseline (FPR 0.364, AUC 0.661). The syntax gradient boosting model (Run 2) also showed a better AUC (0.616) than BERTweet.
  • Subtask 2 (6 months): The Voting Classifier (Run 3) outperformed the Task Baseline in most metrics, achieving an AUC of 0.720 compared to the baseline's 0.764. However, it also had a higher FPR (0.400) than the baseline (0.333). The gradient boosting models performed less favorably in this subtask.
  • BERTweet Limitations: BERTweet was only applied to Subtask 1 due to its maximum sequence length limitations (512 and 128 tokens), which couldn't accommodate the 6 months of tweets in Subtask 2.

Limitations and Future Work

  • Dataset Size and Sequence Length: The small size of the dataset and the inherent sequence length limitations of Transformer models like BERTweet posed significant challenges, leading to overfitting and difficulty in leveraging the full potential of deep learning.
  • Data Aggregation: Aggregating all tweets for a user into a single "document" for classification, especially for longer timeframes, can lead to a loss of temporal information and context.
  • Future Work: The authors suggest exploring alternate data transformations, such as creating tweet-level classification tasks instead of user-level aggregation, to better suit complex models. They also believe that with more time and exploration, variations of Transfer Learning models could be built and leveraged effectively for this task.

Summary updated

BibTeX
@inproceedings{morales2021team,
  author = {Michelle Morales and Prajjalita Dey and Kriti Kohli},
  title = {{Team 9: A Comparison of Simple vs. Complex Models for Suicide Risk Assessment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure