Enhancing personalized suicide risk prediction for VA patients by integrating discrete natural language processing models

Monica Dimambro, Joshua Levy, Jiang Gui, Matan Goldberg, Brian Shiner, Maxwell LevisTranslational 2026 · DOI

TL;DR

This paper tackles the challenge of improving suicide risk prediction for Veterans by integrating two distinct Natural Language Processing (NLP) techniques—semantic and count-based—into a unified model. By weighting these approaches, they achieve enhanced predictive accuracy, particularly for lower-risk patient groups who are often underserved by current methods.

What Problem Does This Paper Address?

Identifying individuals at risk of suicide is a critical public health challenge, especially within the Veteran population, which experiences a disproportionately high suicide rate. The U.S. Department of Veterans Affairs (VA) developed REACH-VET, a model that uses structured electronic health record (EHR) data to predict suicide risk. However, structured data alone has limitations. Unstructured data, such as free-text clinical notes, contains rich psychosocial information that could significantly improve prediction accuracy. Previous research has shown that NLP can extract valuable insights from this unstructured data, but there's a need to effectively combine different NLP methodologies to maximize their predictive power, especially for patient subgroups that are less well-understood and historically underserved by risk prediction models.

Methods

The researchers employed two primary NLP approaches to analyze unstructured EHR notes:

  • Semantic (Closed-Vocabulary) Method: This approach uses theory-based dictionaries and rule-based systems (like SÉANCE) to identify and quantify specific psychosocial variables. It focuses on pre-defined groups of words theoretically linked to concepts like emotions, relationships, or cognitive states.
  • Count (Open-Vocabulary) Method: This approach uses techniques like Count Vectorization to analyze the frequency and distribution of words (unigrams and bigrams) in the text. It allows patterns to emerge directly from the data without pre-defined theoretical assumptions.

They then developed eXtreme Gradient Boosting (XGBoost) classification models. Crucially, they created hybrid models that combined both semantic and count-based features. To balance the influence of these two types of features, they introduced a weighting strategy. This strategy adjusted the probability of selecting semantic versus count-based features during model training, allowing them to fine-tune the model's reliance on each approach based on the risk-tier. Models were evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC), a standard metric for binary classification performance.

Datasets

The study utilized a dataset of Veterans Affairs (VA) patients.

  • Cases: Veterans who died by suicide in 2017-2018 (n=2842).
  • Controls: Risk-matched living Veterans from the same period and facilities (n=14042).

The controls were matched to cases based on their REACH-VET risk percentile at the time of the case's death. The data for analysis was extracted from EHR notes within 30 days prior to the case's death (with notes from 0-5 days excluded to avoid potential endogeneity). This dataset is a subset of the VA Corporate Data Warehouse (CDW) and the VA-Department of Defense Mortality Data Repository (MDR). No new benchmarks were introduced; rather, the study aimed to improve upon the existing REACH-VET benchmark.

Key Insights and Ideas

  • Synergy is Key: Combining semantic and count-based NLP approaches in a weighted manner generally outperforms using either method in isolation, especially for lower-risk patient groups.
  • Lower-Risk Patients Benefit Most: The most significant improvements in predictive accuracy were observed for patients stratified into low and moderate suicide risk tiers. This suggests that current models may be less effective for these populations, and NLP integration can help bridge this gap.
  • Risk-Tier Specificity Matters: The optimal weighting (represented by 'beta') between semantic and count features varied across risk tiers, indicating that different NLP approaches are more influential for different patient groups. For instance, semantic features played a more significant role in the high-risk tier's joint model.
  • Count Models Dominate for Lower Tiers: For moderate and low-risk tiers, count-based models (both alone and in joint models) consistently outperformed semantic-only models, suggesting that the broader, data-driven approach is more effective when clinical notes are sparser or less detailed.
  • Addressing Underserved Populations: The research highlights the potential of NLP to improve risk prediction for patient populations that are poorly understood and frequently underserved by existing models, such as lower-risk Veterans.

Key Results

  • The developed models, particularly the joint (count + semantic) models, demonstrated improved predictive accuracy over the baseline REACH-VET model (AUC > 0.50).
  • For the moderate-risk tier, the best joint model achieved an AUC of 0.66, and for the low-risk tier, it was 0.69. These represent moderate incremental discrimination over the REACH-VET benchmark.
  • Across all risk tiers, count-based models generally outperformed semantic-only models.
  • The optimal weighting (beta) for semantic features was significantly higher in the high-risk tier (0.38) compared to the moderate and low-risk tiers (0.03), indicating a greater influence of semantic variables in the high-risk group's joint model.

Limitations and Future Work

Limitations:

  • Retrospective Data: The study used retrospective data, which may limit generalizability to prospective prediction.
  • VA-Specific Data: The reliance on VA data means findings may not be directly generalizable to non-VA populations.
  • Limited Access to Non-VA Notes: The absence of non-VA provider notes might contribute to the lack of documentation in lower-risk tiers.
  • Provider Bias: The data originates from provider notes, potentially introducing provider biases.
  • Count Model Context: Count-based NLP captures term frequency but limited context, which could affect predictive power.
  • Time Window: The restriction to 5-30 days of EHR notes might miss relevant information if a different window were used.
  • Model Overfitting Risk: The inclusion of many variables and feature weighting without dimensionality reduction could pose a risk of overfitting.
  • High-Risk Tier Benefit: The study offered less benefit for high-risk patients compared to lower-risk tiers, suggesting that alternative machine learning methods or more granular sub-risk tier characterization might be needed for this group.

Future Work:

  • Broader Validation: Validate the methodology in prospective settings and across different healthcare systems.
  • Alternative Models: Compare XGBoost with other analytical tools and model classes (e.g., regularized logistic regression).
  • Enhanced Personalization: Incorporate additional data types like service utilization, medication usage, and psychosocial care.
  • Advanced NLP: Explore deep learning analytic pipelines, including transformers and large language models, for more sophisticated risk analysis.
  • Interactions: Investigate feature interactions within the joint matrix to better understand underlying populations and tailor models.

Summary updated

Abstract

Abstract To improve the identification of Veterans at risk for suicide, the U.S. Department of Veterans Affairs (VA) developed REACH-VET, a suicide risk classification metric. Our previous work demonstrated that incorporating natural language processing (NLP) and developing targeted models for distinct suicide risk-tiers could enhance REACH-VET’s predictive accuracy. This study evaluates the benefits of two NLP methods and compares their predictive performance across risk-tiers. We created a sample of VA patients who either died by suicide in 2017–2018 (cases) or remained alive during that period (controls), stratified by suicide risk (high, moderate, low). We analyzed unstructured electronic health record (EHR) notes using two NLP models: 1) theory-based, closed-vocabulary “semantic” methods, and 2) data-driven, open-vocabulary “count” methods. We then developed eXtreme Gradient Boosting (XGBoost) classification models using semantic, count, and hybrid (count and semantic) variables and calculated area under the receiver operating characteristic curve to assess predictive accuracy. Generally, classification models using semantic variables performed worse than count or hybrid variables. The highest added benefit was seen for low- and moderate-risk patients who achieved incremental improvements in performance over and above leading predictive benchmarks. By using different NLP techniques on unstructured EHR data, our approach improves predictive accuracy for lower-risk patients, expanding suicide prediction for patient populations who are poorly understood and frequently underserved.
BibTeX
@inproceedings{dimambro2026enhancing,
  author = {Monica Dimambro and Joshua Levy and Jiang Gui and Matan Goldberg and Brian Shiner and Maxwell Levis},
  title = {{Enhancing personalized suicide risk prediction for VA patients by integrating discrete natural language processing models}},
  booktitle = {Translational},
  year = {2026},
  doi = {10.1038/s41398-026-03940-8},
}
Figure