Low Resource ICD Coding of Hospital Discharge Summaries

Ashton Williamson, David de Hilster, Amnon Meyers, Nina Hubig, Amy AponACL 2024

TL;DR

This paper introduces a novel, knowledge-based system for automatically assigning ICD-9 codes to hospital discharge summaries, specifically designed to perform well even with very limited or no labeled training data. By leveraging the Unified Medical Language System (UMLS) and a custom NLP++ pipeline, the system achieves competitive performance on low-resource datasets, offering a promising alternative to data-hungry deep learning models.

What Problem Does This Paper Address?

Medical coding, the process of assigning standardized codes to patient health records, is crucial for healthcare analytics, billing, and quality assessment. However, it's a complex, manual task prone to errors, leading to significant financial losses due to improper payments. While Natural Language Processing (NLP) has been applied to automate this, existing machine learning approaches struggle with "low-resource" scenarios – where codes are assigned infrequently, meaning there's little to no labeled training data available. This is a major bottleneck, as collecting and annotating such data is expensive and time-consuming. The paper aims to bridge this gap by developing an automated ICD coding system that doesn't rely on extensive labeled datasets.

Methods

The core of the proposed system is a knowledge-based approach implemented using the NLP++ programming language. The pipeline consists of three main stages:

  1. Extraction: Medical entities are extracted from the input discharge summary.
  2. Linking: These extracted entities are mapped to concepts within the Unified Medical Language System (UMLS). This involves:
  3. Term Normalization: ICD-9 code titles are broken down and mapped to alphanumeric lexical identifiers (EUIs) in the Specialist Lexicon.
  4. Variant Generation: The Lexical Variant Generator (LVG) is used to create a comprehensive list of alternative forms for these terms (abbreviations, acronyms, plural forms, synonyms, etc.). These variants are then organized into knowledge bases.
  5. Ranking: The identified UMLS concepts are used to assign ranking scores to ICD-9 codes. This is achieved using a modified term-frequency inverse-document-frequency (TF-IDF) approach. The system calculates the weight of each ICD-9 code based on the specificity of its constituent terms and then ranks codes based on how well their terms match the input document. The ranking can be performed at the sentence, section, or full note level, with different aggregation methods (max, mean, sum) explored.

A key aspect is the system's ability to function without labeled training data, relying instead on the rich, pre-existing knowledge within the UMLS.

Datasets

The primary dataset used for evaluation is MIMIC-III (Medical Information Mart for Intensive Care). Specifically, the authors utilize:

  • Full MIMIC-III dataset: For general evaluation.
  • Top 50 split: A subset containing only the 50 most frequently assigned codes.
  • Rare 50 split: A newly defined benchmark subset by the authors, focusing on codes that occur fewer than 6 times in the MIMIC-III dataset. This split is crucial for evaluating the system's performance in low-resource settings.

Key Insights and Ideas

  • Knowledge is Power (Even Without Labels): The paper strongly advocates for the utility of knowledge-based systems for ICD coding, demonstrating that leveraging existing biomedical ontologies like UMLS can yield competitive results even in low-resource scenarios where deep learning models falter.
  • The Importance of Lexical Variants: Normalizing terms and generating a wide range of variants (synonyms, abbreviations, etc.) is critical for accurately linking text to medical concepts. This comprehensive approach helps capture more relevant information.
  • Granularity Matters: Ranking codes at a more granular level (sentence or section) rather than the entire document can lead to better performance, suggesting that local context is highly informative for accurate coding.
  • Explainability is Built-in: The knowledge-based nature of the system inherently provides traceability. The system can point to specific terms in the text that support a particular code assignment, a significant advantage for clinical adoption.

Key Results

  • On the Rare 50 split, the proposed LexSyn-Sectionmax system achieved performance comparable to recent state-of-the-art deep learning models, falling within 4 points of the best-performing model in terms of ROC AUC.
  • For the Top 50 split, the system also showed reasonable performance, though it was outpaced by deep learning methods on this less challenging, high-resource dataset.
  • The authors highlight that their system requires no training data, a significant advantage over supervised deep learning approaches.
  • Qualitative analysis revealed that for certain low-performing codes, the system struggled because the relevant terms or concepts were not explicitly present in the text or required inferring from non-textual data (e.g., lab results).

Limitations and Future Work

  • Performance in High-Resource Settings: While competitive in low-resource settings, the system's performance lags behind state-of-the-art deep learning models in high-resource scenarios (like the Top 50 split). More work is needed to improve its performance in these domains.
  • Domain Specificity: The current evaluation is based on a single hospital system's data. Broader evaluation on data from diverse sources is necessary.
  • Word Sense Disambiguation (WSD): The system currently doesn't differentiate between semantic interpretations of medical entities, especially for abbreviations with multiple meanings. Future work could incorporate WSD techniques to improve disambiguation.
  • Leveraging Ontological Information: Further exploration of how to better utilize hierarchical relationships and free-text descriptions within ontologies could enhance performance.
  • Assistive Software: The authors suggest the system could be valuable as assistive software for human coders, flagging inconsistencies or acting as a "first pass" rather than a full replacement.
  • Data Access: Limited access to openly available annotated medical notes remains a challenge for robust evaluation and development.

Summary updated

BibTeX
@inproceedings{williamson2024low,
  author = {Ashton Williamson and David de Hilster and Amnon Meyers and Nina Hubig and Amy Apon},
  title = {{Low Resource ICD Coding of Hospital Discharge Summaries}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure