Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations

Rebecca A. Bernert, Amanda Hilberg, Ruth Melia, Jane Kim, Nigam H. Shah, Freddy AbnousiInternational Journal of Environmental Research and Public Health 2020 · DOI

TL;DR

This systematic review of 87 machine learning (ML) studies reveals that AI/ML approaches show significant promise in predicting suicidal behaviors, achieving high accuracy rates (often >90%) and Area Under the Curve (AUC) values. While the field is rapidly growing, standardization in reporting and methodology is needed to fully realize AI's potential in suicide prevention.

What Problem Does This Paper Address?

Suicide is a major public health crisis, characterized by its complex, transdiagnostic nature and its relative rarity at the population level, making prediction extremely challenging. Despite existing prevention strategies, suicide rates remain stubbornly high and have even increased in some regions. Traditional risk assessment methods often lack sensitivity and clinical utility. This paper addresses the need for more effective suicide risk detection and prevention by systematically reviewing how Artificial Intelligence (AI) and Machine Learning (ML) are being applied to large datasets to improve prediction accuracy.

Methods

The authors conducted a systematic literature search across multiple databases (PubMed/MEDLINE, PsychInfo, Web-of-Science, EMBASE) using keywords related to AI, machine learning, and suicide. They supplemented this with hand-searching and Google Scholar. Inclusion criteria focused on original journal articles in English that employed AI/ML methodologies to evaluate suicide risk outcomes. A PRISMA flow diagram was used to track the study selection process. Data extraction involved coding for study characteristics, ML methodology (supervised vs. unsupervised learning, specific algorithms), and performance metrics (accuracy, AUC, sensitivity, specificity). Descriptive analyses were used to synthesize the findings, as meta-analysis was not possible due to methodological heterogeneity.

Datasets

The paper does not introduce new datasets. Instead, it reviews studies that utilized various existing datasets, including:

  • Electronic Medical Records (EMR)
  • Administrative chart data
  • Epidemiologic surveys
  • Social media data (e.g., Twitter, microblogs)
  • Clinical samples (e.g., from high-risk or triage settings like emergency departments)
  • Archival datasets

The total patient sample size across the included studies was substantial, aggregating to 5,986,238 patients, though sample size was unreported in 7 studies.

Key Insights and Ideas

  • AI/ML is a rapidly growing field for suicide prevention: Nearly half of the reviewed studies were published in the last three years, indicating a significant surge in research interest.
  • High predictive performance is achievable: Many studies reported impressive accuracy (>90%) and AUC values, suggesting ML models can effectively identify individuals at risk.
  • Supervised learning dominates: The majority of studies employed supervised learning techniques, with ensemble methods, naive Bayes, decision trees, and SVMs being common. Unsupervised learning was used less frequently.
  • NLP offers unique insights: Natural Language Processing (NLP) was employed in 29 investigations, particularly for analyzing text from social media and clinical notes, showing promise in detecting subtle linguistic cues related to suicide risk.
  • Transdiagnostic approach is prevalent: Most studies focused on a broad range of individuals rather than specific psychiatric conditions, reflecting the transdiagnostic nature of suicide.
  • Data sources are diverse: ML models are being trained on a wide array of data, from structured EMR data to unstructured social media posts, highlighting the versatility of AI.

Key Results

  • 87 studies were included in the qualitative synthesis, with 13 forming a subset focused on NLP analysis of suicide notes.
  • High performance metrics: For classification studies (n=65), the average AUC was 0.814 (SD=0.110) and average accuracy was 0.813 (SD=0.123).
  • Outcome variability: Studies assessed various suicide outcomes, including suicide attempt (n=28), suicide death (n=14), suicidal ideation (n=9), multiple risk outcomes (n=18), and other-social media (n=10) or undifferentiated outcomes (n=8).
  • No significant difference in performance by outcome type: An exploratory ANOVA found no significant differences in accuracy or AUC based on the broad suicide outcome groupings.
  • NLP for suicide notes: A subset of 13 studies using NLP on suicide decedent notes showed promising results, with precision values often above 0.60.

Limitations and Future Work

  • Methodological heterogeneity: The review highlighted significant variation in ML methods, study quality, and outcome definitions, making direct comparisons and meta-analysis challenging.
  • Underreporting of key metrics: Parameters like Positive Predictive Value (PPV), Area Under the Precision Curve (AUPRC), and lead-time of prediction were often underreported.
  • Cross-sectional designs dominate: The majority of studies were cross-sectional, underscoring the need for more prospective, longitudinal designs to advance prediction and intervention.
  • Generalizability concerns: While many studies used large datasets, concerns about overfitting exist for studies with high-dimensional data and small sample sizes. Replication in new settings and populations is crucial.
  • Data limitations: Challenges with EMR data (e.g., incomplete records, biases in collection) and the low correspondence between ICD codes and clinical notes for suicide risk were noted.
  • Future directions: The authors emphasize the need for improved reporting standards, a priori-designed studies, replication of successful models, integration of ML into clinical workflows, and a focus on clinical need to guide ML applications. They also suggest exploring the combination of biological, behavioral, and clinical variables for a more comprehensive approach.

Summary updated

Abstract

Suicide is a leading cause of death that defies prediction and challenges prevention efforts worldwide. Artificial intelligence (AI) and machine learning (ML) have emerged as a means of investigating large datasets to enhance risk detection. A systematic review of ML investigations evaluating suicidal behaviors was conducted using PubMed/MEDLINE, PsychInfo, Web-of-Science, and EMBASE, employing search strings and MeSH terms relevant to suicide and AI. Databases were supplemented by hand-search techniques and Google Scholar. Inclusion criteria: (1) journal article, available in English, (2) original investigation, (3) employment of AI/ML, (4) evaluation of a suicide risk outcome. N = 594 records were identified based on abstract search, and 25 hand-searched reports. N = 461 reports remained after duplicates were removed, n = 316 were excluded after abstract screening. Of n = 149 full-text articles assessed for eligibility, n = 87 were included for quantitative synthesis, grouped according to suicide behavior outcome. Reports varied widely in methodology and outcomes. Results suggest high levels of risk classification accuracy (>90%) and Area Under the Curve (AUC) in the prediction of suicidal behaviors. We report key findings and central limitations in the use of AI/ML frameworks to guide additional research, which hold the potential to impact suicide on broad scale.
BibTeX
@article{bernert2020artificial,
  author = {Rebecca A. Bernert and Amanda Hilberg and Ruth Melia and Jane Kim and Nigam H. Shah and Freddy Abnousi},
  title = {{Artificial Intelligence and Suicide Prevention: A Systematic Review of Machine Learning Investigations}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2020},
  doi = {10.3390/ijerph17165929},
}
Figure