Introduction
Suicide remains a critical global public health crisis. In clinical settings, identifying patients at risk is paramount for intervention. However, current methods relying on structured data, such as International Classification of Diseases (ICD) codes, are often inadequate. Research indicates that ICD codes capture only a small fraction of suicidal behaviors documented in medical records—approximately 3% of suicide ideation (SI) events and 19% of suicide attempts (SA). The "ground truth" regarding a patient's mental state is frequently buried within unstructured Electronic Health Record (EHR) narratives, such as nursing notes and discharge summaries.
To address this gap, researchers presented ScAN (Suicide Attempt and Ideation Events Dataset) and ScANER, a specialized NLP framework, at NAACL 2022. ScAN represents the largest publicly available dataset of annotated SA and SI events, providing the granularity needed for effective clinical modeling.
The ScAN Dataset
The ScAN dataset is a curated subset of the MIMIC-III database, comprising 12,759 clinical notes across 697 hospital stays. It contains 19,690 unique evidence annotations. Unlike previous datasets that often relied on binary classification, ScAN introduces nuanced categories:
- Positive: Explicit evidence of SA or SI.
- Negative: Explicit documentation that a patient denies suicidal thoughts or that an event was not a suicide attempt.
- Unsure: Used when intent is ambiguous, such as a drug overdose where the patient's motivation is not yet documented.
Furthermore, the dataset maps suicide attempts to specific ICD-10-CM groups, such as poisoning or asphyxiation, adding a layer of clinical utility missing from broader datasets.
The ScANER Framework
Alongside the dataset, the authors developed ScANER (Suicide Attempt and Ideation Events Retriever), a two-stage pipeline designed to identify suicidal behaviors at the hospital-stay level.
- Evidence Retriever: This module utilizes medRoBERTa, a version of the RoBERTa model pre-trained on the MIMIC-III clinical corpus. It processes notes at the paragraph level to identify evidence of SA or SI. To enhance robustness, the model is trained using multi-task learning, simultaneously classifying the behavior type while retrieving evidence.
- Predictor Module: Snippets identified by the retriever are aggregated by a multi-head attention model. This model makes a final "stay-level" prediction, determining if the entire hospital visit indicates a risk of SA or SI.
To handle the inherent challenges of EHR data, such as extreme class imbalance and clinical noise, the team employed weighted log loss and robust fine-tuning techniques.
Results and Clinical Insights
The ScANER framework prioritizes high recall, reflecting the clinical reality that missing a high-risk patient (a false negative) is far more dangerous than a false positive.
- Evidence Extraction: The retriever achieved a macro-weighted F1-score of 0.83.
- Stay-Level Prediction: For suicide attempts, the model reached an F1 of 0.78 with a recall of 0.81. For suicide ideation, it achieved an F1 of 0.60.
- Domain Adaptation: The use of medRoBERTa proved crucial, as it consistently outperformed general-purpose models like clinicalBERT, highlighting the importance of domain-specific pre-training for clinical NLP tasks.
A key insight from the study is the necessity of the "Unsure" category. By acknowledging medical ambiguity, the model aligns more closely with real-world physician workflows. Additionally, the study confirms that suicidality information is distributed across various note types, necessitating an aggregate view of the patient's stay.
Limitations and Future Directions
While ScAN and ScANER mark a significant advancement, several challenges remain. The dataset contains relatively few "Negative SA" cases, making it difficult for models to learn patterns where self-harm is explicitly ruled out as a suicide attempt. Furthermore, the model occasionally struggles with temporal nuance—distinguishing between historical suicidal thoughts and current crises. Future work will focus on expanding the dataset with more negative instances and refining the model's ability to interpret the timeline of a patient's mental health history.
Reference: Rawat, B. P. S., Kovaly, S., Pigeon, W. R., & Yu, H. (2022). ScAN: Suicide Attempt and Ideation Events Dataset. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Summary updated