TL;DR
This paper tackles the challenge of improving suicide risk prediction for Veterans by integrating two distinct Natural Language Processing (NLP) techniques—semantic and count-based—into a unified model. By weighting these approaches, they achieve enhanced predictive accuracy, particularly for lower-risk patient groups who are often underserved by current methods.
What Problem Does This Paper Address?
Identifying individuals at risk of suicide is a critical public health challenge, especially within the Veteran population, which experiences a disproportionately high suicide rate. The U.S. Department of Veterans Affairs (VA) developed REACH-VET, a model that uses structured electronic health record (EHR) data to predict suicide risk. However, structured data alone has limitations. Unstructured data, such as free-text clinical notes, contains rich psychosocial information that could significantly improve prediction accuracy. Previous research has shown that NLP can extract valuable insights from this unstructured data, but there's a need to effectively combine different NLP methodologies to maximize their predictive power, especially for patient subgroups that are less well-understood and historically underserved by risk prediction models.
Methods
The researchers employed two primary NLP approaches to analyze unstructured EHR notes:
- Semantic (Closed-Vocabulary) Method: This approach uses theory-based dictionaries and rule-based systems (like SÉANCE) to identify and quantify specific psychosocial variables. It focuses on pre-defined groups of words theoretically linked to concepts like emotions, relationships, or cognitive states.
- Count (Open-Vocabulary) Method: This approach uses techniques like Count Vectorization to analyze the frequency and distribution of words (unigrams and bigrams) in the text. It allows patterns to emerge directly from the data without pre-defined theoretical assumptions.
They then developed eXtreme Gradient Boosting (XGBoost) classification models. Crucially, they created hybrid models that combined both semantic and count-based features. To balance the influence of these two types of features, they introduced a weighting strategy. This strategy adjusted the probability of selecting semantic versus count-based features during model training, allowing them to fine-tune the model's reliance on each approach based on the risk-tier. Models were evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC), a standard metric for binary classification performance.
Datasets
The study utilized a dataset of Veterans Affairs (VA) patients.
- Cases: Veterans who died by suicide in 2017-2018 (n=2842).
- Controls: Risk-matched living Veterans from the same period and facilities (n=14042).
The controls were matched to cases based on their REACH-VET risk percentile at the time of the case's death. The data for analysis was extracted from EHR notes within 30 days prior to the case's death (with notes from 0-5 days excluded to avoid potential endogeneity). This dataset is a subset of the VA Corporate Data Warehouse (CDW) and the VA-Department of Defense Mortality Data Repository (MDR). No new benchmarks were introduced; rather, the study aimed to improve upon the existing REACH-VET benchmark.
Key Insights and Ideas
- Synergy is Key: Combining semantic and count-based NLP approaches in a weighted manner generally outperforms using either method in isolation, especially for lower-risk patient groups.
- Lower-Risk Patients Benefit Most: The most significant improvements in predictive accuracy were observed for patients stratified into low and moderate suicide risk tiers. This suggests that current models may be less effective for these populations, and NLP integration can help bridge this gap.
- Risk-Tier Specificity Matters: The optimal weighting (represented by 'beta') between semantic and count features varied across risk tiers, indicating that different NLP approaches are more influential for different patient groups. For instance, semantic features played a more significant role in the high-risk tier's joint model.
- Count Models Dominate for Lower Tiers: For moderate and low-risk tiers, count-based models (both alone and in joint models) consistently outperformed semantic-only models, suggesting that the broader, data-driven approach is more effective when clinical notes are sparser or less detailed.
- Addressing Underserved Populations: The research highlights the potential of NLP to improve risk prediction for patient populations that are poorly understood and frequently underserved by existing models, such as lower-risk Veterans.
Key Results
- The developed models, particularly the joint (count + semantic) models, demonstrated improved predictive accuracy over the baseline REACH-VET model (AUC > 0.50).
- For the moderate-risk tier, the best joint model achieved an AUC of 0.66, and for the low-risk tier, it was 0.69. These represent moderate incremental discrimination over the REACH-VET benchmark.
- Across all risk tiers, count-based models generally outperformed semantic-only models.
- The optimal weighting (beta) for semantic features was significantly higher in the high-risk tier (0.38) compared to the moderate and low-risk tiers (0.03), indicating a greater influence of semantic variables in the high-risk group's joint model.
Limitations and Future Work
Limitations:
- Retrospective Data: The study used retrospective data, which may limit generalizability to prospective prediction.
- VA-Specific Data: The reliance on VA data means findings may not be directly generalizable to non-VA populations.
- Limited Access to Non-VA Notes: The absence of non-VA provider notes might contribute to the lack of documentation in lower-risk tiers.
- Provider Bias: The data originates from provider notes, potentially introducing provider biases.
- Count Model Context: Count-based NLP captures term frequency but limited context, which could affect predictive power.
- Time Window: The restriction to 5-30 days of EHR notes might miss relevant information if a different window were used.
- Model Overfitting Risk: The inclusion of many variables and feature weighting without dimensionality reduction could pose a risk of overfitting.
- High-Risk Tier Benefit: The study offered less benefit for high-risk patients compared to lower-risk tiers, suggesting that alternative machine learning methods or more granular sub-risk tier characterization might be needed for this group.
Future Work:
- Broader Validation: Validate the methodology in prospective settings and across different healthcare systems.
- Alternative Models: Compare XGBoost with other analytical tools and model classes (e.g., regularized logistic regression).
- Enhanced Personalization: Incorporate additional data types like service utilization, medication usage, and psychosocial care.
- Advanced NLP: Explore deep learning analytic pipelines, including transformers and large language models, for more sophisticated risk analysis.
- Interactions: Investigate feature interactions within the joint matrix to better understand underlying populations and tailor models.
Summary updated