Natural language processing for scalable feature engineering and ultra-high-dimensional confounding adjustment in healthcare database studies

Richard Wyss, Jie Yang, Sebastian Schneeweiß, Joseph M. Plasek, Li Zhou, Thomas Deramus, Janick Weberpals, Kerry Ngan, Theodore Tsacogianis, Kueiyu Joshua Lin

TL;DR

Researchers have demonstrated that using unsupervised Natural Language Processing (NLP) to extract thousands of features from "free-text" clinical notes significantly improves the accuracy of observational healthcare studies. By supplementing standard insurance claims data with these NLP-generated features, the team achieved better "covariate balance"—essentially making comparison groups more similar—and found that simple techniques like n-grams often perform just as well as complex deep-learning models for this task.

What Problem Does This Paper Address?

In medical research, "real-world evidence" (RWE) often comes from analyzing existing databases like insurance claims or Electronic Health Records (EHR). However, these studies face a massive hurdle: confounding by indication.

Imagine comparing two drugs. If doctors prescribe Drug A only to the sickest patients and Drug B to healthier ones, Drug A will look "worse" simply because of who received it, not because of the drug itself. To fix this, researchers try to adjust for every possible variable (confounder). While structured data (like ICD-10 diagnosis codes) helps, it misses the rich, nuanced information hidden in a doctor’s handwritten notes—things like a patient's frailty, lifestyle, or subtle symptom progression. Manually reading these notes is impossible at scale, leaving a wealth of "proxy" information underutilized.

Methods

The authors developed an analytic pipeline to turn unstructured clinical text into structured "features" that can be fed into statistical models. They compared eight different Propensity Score (PS) models, ranging from those using only researcher-specified variables to those incorporating thousands of automated features.

Key NLP techniques tested included:

  • N-grams (Bag-of-words): Simple counts of single words or two-word sequences.
  • MTERMS: A tool specifically designed to extract clinical concepts (problems, meds, allergies).
  • Word/Sentence Embeddings (GloVe, BioBERT, BERT): Advanced deep-learning methods that represent words or sentences as mathematical vectors to capture semantic meaning.

The team used LASSO regression to handle the "ultra-high-dimensional" nature of the data (often over 30,000 candidate predictors) and applied Matching Weights to balance the treatment groups.

Datasets

The study utilized a powerful linked dataset:

  • Medicare Claims: Longitudinal insurance data for fee-for-service beneficiaries aged 65+.
  • RPDR (Research Patient Data Registry): Detailed EHR data from Mass General Brigham, including inpatient/outpatient notes, lab results, and procedures from 2007 to 2017.

They evaluated these methods across three distinct clinical cohorts:

  1. PPI Cohort: High-dose vs. low-dose proton pump inhibitors (Risk: GI bleeding).
  2. HTN Cohort: ACE inhibitors vs. Beta-blockers (Risk: Hyperkalemia).
  3. Analgesics Cohort: NSAIDs vs. Opioids (Risk: Acute Kidney Injury).

Key Insights and Ideas

  • The "Simplicity" Surprise: One of the most striking findings was that simple n-grams (Model 4) often outperformed or matched sophisticated deep-learning models like BioBERT in terms of predictive performance and covariate balance. In this context, knowing a specific word appears in a note is often more useful than a complex mathematical representation of that word's "context."
  • Scalability without Supervision: The entire pipeline is unsupervised. It doesn't require doctors to manually label "gold standard" charts, which means this approach can be scaled across millions of patients and different hospital systems rapidly.
  • Capturing the "Unmeasured": The NLP features acted as "proxies" for factors researchers didn't explicitly look for. By including thousands of these proxies, the models were able to "balance" the groups on characteristics that were previously invisible to the researchers.

Key Results

  • Improved Balance: Including NLP features brought the "standardized difference" (a measure of group similarity) to < 0.1 for nearly all variables across all three studies. Without NLP, many important clinical features remained wildly unbalanced.
  • Moving the Needle on Effect Estimates:
    • In the PPI study, the unadjusted Hazard Ratio (HR) was a heavily biased 2.77. Adjusting with n-gram NLP features moved the HR to 1.38 (95% CI: 0.95, 1.97), which aligns much closer to the "null" result expected from randomized controlled trials.
  • In the Analgesics study, the HR moved from 0.41 (unadjusted) to 0.81 after NLP adjustment, successfully correcting for the fact that opioids are usually given to patients with worse kidney function.
  • Predictive Power: Models using NLP features consistently showed higher AUCs (Area Under the Curve), indicating they were much better at identifying why a patient received a specific treatment.

Limitations and Future Work

While successful, the authors noted several limitations:

  • Target Population Shifts: Using matching and overlap weights changes the "target population" to those where treatment equipoise exists. This makes it harder to generalize results to the entire patient population.
  • The "Black Box" of Proxies: While the features improve balance, we don't always know why a specific n-gram is a good proxy, which can be a hurdle for clinical interpretation.
  • LLMs on the Horizon: This study used "off-the-shelf" NLP. The authors suggest that future work should explore Large Language Models (LLMs) to see if they can extract even more nuanced confounder information, though they cautioned that more complexity doesn't always equal better causal inference.

Summary updated

Abstract

Supplementing administrative claims with large numbers of NLP-generated features for ultra-high-dimensional proxy confounder adjustment improved overall covariate balance and may provide a modest benefit in terms of capturing confounder information.
BibTeX
@article{wyss2025natural,
  author = {Richard Wyss and Jie Yang and Sebastian Schneeweiß and Joseph M. Plasek and Li Zhou and Thomas Deramus and Janick Weberpals and Kerry Ngan and Theodore Tsacogianis and Kueiyu Joshua Lin},
  title = {{Natural language processing for scalable feature engineering and ultra-high-dimensional confounding adjustment in healthcare database studies}},
  journal = {Journal of Biomedical Informatics},
  year = {2025},
  doi = {10.1016/j.jbi.2025.104882},
}

Loading related content...

Figure