Using Psychologically-Informed Priors for Suicide Prediction in the CLPsych 2021 Shared Task

Avi Gamoran, Yonatan Kaplan, Almog Simchon, Michael GileadNAACL 2021

TL;DR

This paper tackles the challenging task of predicting suicide attempts using social media data, a domain plagued by scarce labeled examples and a high risk of overfitting. The authors propose a novel approach that leverages psychologically-informed priors within a Bayesian modeling framework, demonstrating that incorporating domain knowledge can significantly improve prediction accuracy and robustness, especially when data is limited.

What Problem Does This Paper Address?

Suicide is a critical public health issue, and early identification of individuals at risk is crucial for prevention. Social media platforms offer a vast, albeit noisy, source of data that could potentially reveal warning signs. However, predicting suicide attempts from this data is fraught with challenges. Firstly, labeled datasets are scarce due to the sensitive nature of the information, leading to a high risk of models overfitting to the limited training data. Secondly, standard machine learning approaches, which rely on learning patterns from data, may struggle to generalize effectively in such noisy and data-limited environments. This paper addresses the need for robust suicide prediction models that can overcome these limitations by effectively integrating existing psychological knowledge with empirical data.

Methods

The core of the authors' approach lies in using Bayesian modeling to incorporate psychologically-informed priors. Instead of relying solely on data-driven inductive learning, they engineered features based on established psychological theories and prior research findings. These priors were then integrated into logistic regression models.

Key methodological aspects include:

  • Theory-Driven Feature Engineering: The authors developed novel features based on psychological theories related to mental vulnerability, depression, and suicide. This included features derived from the Linguistic Inquiry and Word Count (LIWC) dictionary and a custom "Mind-Perception Dictionary."
  • Bayesian Logistic Regression: They employed Bayesian logistic regression models, which allow for the formal integration of prior beliefs (informed by psychological literature) with the observed data. This contrasts with standard frequentist approaches.
  • Informed vs. Weakly-Informed Priors: The study compared models using "informed priors" (derived from effect sizes in previous studies) with "weakly-informed priors" (general priors with minimal assumptions).
  • Regularized Regression: As a baseline and comparison, they also implemented standard regularized regression models (Ridge and Lasso) using the glmnet package.
  • Model Selection: For hyperparameter tuning, they used cross-validation.

Datasets

The study utilized data from the CLPsych 2021 Shared Task. This dataset consists of de-identified Twitter feeds from individuals who have made suicide attempts and control individuals. The task involved predicting suicide attempts based on tweets from two time windows:

  • Subtask 1: Up to 30 days before suicide attempts.
  • Subtask 2: Up to 182 days (6 months) before suicide attempts.

The paper notes the scarcity of training examples for this task (e.g., 114/164 individuals in the current task).

Key Insights and Ideas

  • Domain Knowledge is Crucial for Data-Scarce Problems: The paper strongly suggests that when dealing with limited, sensitive data like suicide attempts on social media, incorporating prior domain knowledge (psychology) is not just beneficial but essential to prevent overfitting and achieve meaningful predictions.
  • Bayesian Priors Mitigate Overfitting: By using psychologically-informed priors, the models were less prone to overfitting the training data, leading to better generalization on unseen test data. This is a significant advantage in domains with small sample sizes.
  • Interpretability and Cumulative Science: The Bayesian approach, particularly with informed priors, enhances model interpretability. The ability to share and update empirical priors facilitates cumulative scientific discovery in the field.
  • False Positives vs. False Negatives: The authors highlight that their Bayesian models were particularly successful in avoiding false positive predictions. While in clinical settings, erring on the side of caution (minimizing false negatives) might be preferred, for broad risk detection on social media, avoiding false alarms is also important to prevent unwarranted alarm and potential suicidal suggestibility.

Key Results

The models generally performed above chance, with notable differences between the subtasks and models:

  • Subtask 1 (30 days): The informed Bayesian model (M1) slightly outperformed the task's baseline. The regularized regression model (M3) achieved the highest AUC (0.599), but the informed model (M1) was more successful in avoiding false positives.
  • Subtask 2 (6 months): The informed Bayesian model (M1) showed considerably better predictions, achieving AUC scores between 0.739 and 0.809. The regularized regression model (M3) achieved the highest AUC (0.844) in this subtask, likely benefiting from the larger dataset.
  • Overall: In both tasks, the Bayesian models (M1 and M2) were particularly effective at minimizing false positive prediction outcomes. The informed Bayesian model (M1) generally outperformed the weakly-informed model (M2) in Subtask 1, and showed strong performance in Subtask 2.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the theory-guided approach increases bias and lowers accuracy on the training set, it was successful in preventing overfitting. This trade-off between bias and variance is inherent in such approaches.
  • The authors note that the regularized regression models (M3) were slightly less accurate in Subtask 1 than the informed model (M1), but slightly more accurate in Subtask 2, possibly due to the larger dataset in Subtask 2.
  • Future Work:
    • The authors suggest that future research should continue to explore how psychological knowledge and data science methods can be combined to build more effective suicide prevention tools.
  • They emphasize the need for actionable predictions that can help save lives.
  • The paper also implicitly points to the need for larger, more diverse datasets to further refine these models.

Summary updated

BibTeX
@inproceedings{gamoran2021using,
  author = {Avi Gamoran and Yonatan Kaplan and Almog Simchon and Michael Gilead},
  title = {{Using Psychologically-Informed Priors for Suicide Prediction in the CLPsych 2021 Shared Task}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure