Natural Language Processing Algorithms Outperform ICD Codes in the Development of Fall Injuries Registry

Atta Taseh, Souri Sasanfar, Jia-Zhen M. Chan, Evan Sirls, Ara Nazarian, Kayhan Batmanghelich, Jonathan F. Bean, Soheil Ashkani‐EsfahanimedRxiv 2024 · DOI

TL;DR

Researchers have demonstrated that Natural Language Processing (NLP) models can identify fall-related injuries in medical records with far greater accuracy than traditional administrative billing codes (ICD codes). While ICD codes caught only 26% of fall incidents, NLP algorithms identified up to 98%, offering a powerful tool for building more accurate and automated disease registries.

What Problem Does This Paper Address?

Falls are a massive public health burden, causing millions of emergency visits and costing an estimated $50 billion annually in the U.S. alone. To prevent future falls, researchers need high-quality "fall registries"—databases that track when, why, and how these injuries happen.

Currently, these registries rely on ICD (International Classification of Diseases) codes. However, there is a major "visibility" gap: doctors often code for the result of a fall (like a hip fracture) rather than the cause (the fall itself). Because reporting the "External Cause of Morbidity" isn't always mandatory, fall data is frequently lost in the shuffle. Manually reading thousands of clinical notes to find this missing data is too slow and expensive. This paper investigates whether NLP can bridge this gap by "reading" unstructured clinical notes to automatically detect falls and their specific mechanisms.

Methods

The researchers conducted a retrospective case-control study using clinical notes from four tertiary hospitals in Boston. They focused on two primary tasks:

  1. Fall Occurrence Detection: A binary classification task (Did a fall happen or not?).
  2. Fall Mechanism Classification: A multi-class task categorizing the fall into three types: Same-level (tripping while walking), Multi-level (falling down stairs or off a chair), or Unclassified.

The team compared several machine learning architectures:

  • BERT (Bidirectional Encoder Representations from Transformers): A deep learning model designed to understand the context of words in a sentence.
  • Ensemble Methods: Algorithms like XGBoost, AdaBoost, and Random Forest (RF), which combine multiple "weak" models to create a stronger predictor.
  • SVM (Support Vector Machine): A classic classifier used for identifying linear boundaries between data groups.

Before training, the notes underwent rigorous preprocessing, including de-identification (removing patient names), segmentation (breaking notes into logical sections), and cleaning.

Datasets

The study utilized a dataset of 1,769 clinical notes retrieved via CPT codes for hip fractures between 2010 and 2019.

  • Fall Occurrence Task: 1,769 notes (791 cases of falls, 978 controls).
  • Fall Mechanism Task: 783 notes (511 same-level, 151 multi-level, and 121 unclassified).

The "ground truth" for these datasets was established through manual annotation by experienced medical experts.

Key Insights and Ideas

  • The ICD "Blind Spot": The most striking insight is the sheer inadequacy of administrative codes. ICD codes missed nearly 75% of fall incidents and almost 90% of fall mechanisms. This suggests that current registries based solely on billing data are significantly undercounting the problem.
  • Ensembles vs. Transformers: Interestingly, while BERT is often the "gold standard" for NLP, ensemble methods like AdaBoost and XGBoost actually outperformed or matched it in fall detection. The authors suggest that BERT’s performance is highly dependent on large volumes of training data; for niche medical tasks with smaller datasets, traditional machine learning with smart feature engineering (like TF-IDF) remains highly competitive.
  • The "Same-Level" Bias: The models were significantly better at identifying same-level falls than multi-level ones. This reflects a real-world data imbalance: most elderly patients fall from a standing position.

Key Results

The quantitative gap between NLP and traditional methods was substantial:

  • Detection Accuracy: NLP models (AdaBoost/XGBoost) correctly identified 98% of fall occurrences, compared to just 26% for ICD codes.
  • Detection Performance: The best NLP model for fall detection achieved an F1-score of 0.97 and an AUC-ROC of 0.99, indicating near-perfect classification.
  • Mechanism Accuracy: NLP models correctly classified the way a person fell in 65% of cases, whereas ICD codes only managed 12%.
  • Mechanism Performance: The Random Forest model was the top performer for classifying mechanisms, with an AUC-ROC of 0.70.

Limitations and Future Work

The authors identify a few hurdles that still need to be cleared:

  • Class Imbalance: Because "same-level" falls are so common, the models struggled to accurately classify the rarer "multi-level" falls. Future work will require larger, more granular datasets to train models on these less frequent scenarios.
  • Note Variability: Clinical notes are written by different professionals with varying styles. While the NLP handled this well, the "unclassified" category (where notes lacked sufficient detail) remains a challenge that no algorithm can solve without better documentation at the point of care.
  • Scalability: While promising, the study was limited to hip fracture patients. Expanding this to other injury types would further validate the approach for a universal fall registry.

Summary updated

Abstract

Abstract Background Standardized registries are commonly built using administrative codes assigned to patient encounters, such as the International Classification of Diseases (ICD) codes. However, fall patients are often coded using subsequent injury codes, such as hip fractures. This necessitates manual screening to ensure the accuracy of data registries. Herein, we aimed to automate the extraction of fall incidents and mechanisms using Natural Language Processing (NLP) and compare this approach with the ICD method. Methods Clinical notes for patients with fall-induced hip fractures were retrospectively reviewed by medical experts. Fall incidences were detected, annotated, and classified among patients who had fall-induced hip fracture (case group). The control group included patients with hip fracture without any evidence of fall. NLP models were developed using the annotated notes of the study groups to fulfill two separate tasks: fall occurrence detection and fall mechanism classification. The performances of the models were compared using accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F1-score, and area under the ROC curve (AUC-ROC). Results A total of 1,769 clinical notes were included in the final analysis for the fall occurrence task, and 783 clinical notes were analyzed for the fall mechanism classification task. The highest F1 score using NLP for fall occurrence was 0.97 (specificity=0.96; sensitivity=0.97) and for fall mechanism classification was 0.61 (specificity=0.56; sensitivity=0.62). NLP could detect up to 98% of the fall occurrences and 65% of the fall mechanisms accurately compared to 26% and 12%, respectively, by ICD codes. Conclusion Our findings showed promising performance with a higher accuracy of NLP algorithms compared to the conventional method for detecting fall occurrence and mechanism in developing disease registries using clinical notes. Our approach can be introduced to other registries that are based on large data and are in need for accurate annotation and classification.
BibTeX
@misc{taseh2024natural,
  author = {Atta Taseh and Souri Sasanfar and Jia-Zhen M. Chan and Evan Sirls and Ara Nazarian and Kayhan Batmanghelich and Jonathan F. Bean and Soheil Ashkani‐Esfahani},
  title = {{Natural Language Processing Algorithms Outperform ICD Codes in the Development of Fall Injuries Registry}},
  year = {2024},
  doi = {10.1101/2024.09.26.24314444},
  howpublished = {medRxiv},
}
Figure