A Label Attention Model for ICD Coding from Clinical Text

Thanh Vu, Dat Quoc Nguyen, Anthony N. NguyenIJCAI 2020 · DOI

TL;DR

This paper introduces a novel "Label Attention Model" (LAAT) for automatically assigning International Classification of Diseases (ICD) codes from clinical text. To address the challenge of imbalanced data (many rare codes), they also propose a hierarchical joint learning mechanism that leverages the inherent structure of ICD codes, achieving state-of-the-art results on benchmark datasets.

What Problem Does This Paper Address?

Assigning ICD codes to clinical notes is a crucial but labor-intensive process for healthcare. Manual coding is prone to errors and requires significant human resources. While machine learning has been applied to automate this, existing methods often struggle with:

  • Variable Text Fragment Importance: ICD codes can be linked to text fragments that are far apart in a clinical note, making fixed-size window approaches (like traditional CNNs) insufficient.
  • Data Imbalance: The vast majority of ICD codes are rarely used, leading to highly imbalanced datasets where models tend to perform poorly on infrequent diagnoses.

This paper aims to overcome these limitations by developing a more sophisticated attention mechanism and a hierarchical learning approach.

Methods

The core of the proposed solution is a Label Attention Model (LAAT). Its architecture consists of:

  1. Embedding Layer: Standard word embeddings are used to represent input tokens.
  2. Bidirectional LSTM (BiLSTM) Layer: This layer captures contextual information from the clinical text by processing it in both forward and backward directions, generating latent feature vectors for each word.
  3. Label Attention Layer: This is the novel component. It takes the BiLSTM outputs and learns label-specific attention weights. Instead of a single attention mechanism for the whole document, it generates a unique representation for the document tailored to each specific ICD code. This allows the model to focus on the most relevant text fragments for each individual code.
  4. Output Layer: For each label-specific vector, a feed-forward network with a sigmoid activation function predicts the probability of that ICD code being present.

To tackle data imbalance, they introduce a Hierarchical Joint Learning Mechanism. This approach leverages the hierarchical structure of ICD codes (e.g., codes starting with the same three characters belong to a higher-level category). The model first predicts a "normalized" code (the first three characters) and then uses this prediction to inform the prediction of the more specific "raw" ICD code. This allows the model to learn from the relationships between codes, improving performance on rare ones.

Datasets

The models were evaluated on three benchmark datasets commonly used in ICD coding research:

  • MIMIC-III: This is the primary dataset, used in both full (MIMIC-III-full) and a subset of the 50 most frequent codes (MIMIC-III-50) settings. Discharge summaries are used.
  • MIMIC-II: Used in a full code setting (MIMIC-II-full). Clinical notes are used.

The paper follows standard data splitting protocols based on patient IDs to ensure no data leakage between training, validation, and testing sets.

Key Insights and Ideas

  • Label-Specific Attention is Crucial: The paper highlights that a single, aggregated text representation is insufficient for multi-label tasks like ICD coding. Tailoring attention to each label allows the model to effectively capture context for diverse diagnoses.
  • Hierarchical Structure Aids Rare Codes: Leveraging the inherent hierarchy of ICD codes is a powerful strategy to mitigate the impact of data imbalance, leading to better performance on less frequent diagnoses.
  • BiLSTM Outperforms BiGRU: The study found that BiLSTM, with its separate memory cells, is more effective than BiGRU for processing long clinical texts, likely due to its better ability to capture long-term dependencies.

Key Results

The proposed LAAT model achieved state-of-the-art (SOTA) results on the MIMIC-III-full and MIMIC-II-full datasets across various metrics, including macro- and micro-averaged F1 scores and AUC.

  • MIMIC-III-full: LAAT improved macro-F1 by 2.2% and micro-F1 by 0.9% over previous SOTA models like MSATT-KG and MultiResCNN.
  • MIMIC-III-50: LAAT also outperformed previous SOTA models, showing notable improvements in macro-F1 and micro-F1.
  • MIMIC-II-full: LAAT significantly improved micro-F1 by 12.5% over HA-GRU and showed substantial gains in other metrics compared to MultiResCNN.
  • JointLAAT (Hierarchical Model): For datasets with infrequent codes (MIMIC-III-full and MIMIC-II-full), JointLAAT demonstrated significant improvements in macro-averaged metrics (e.g., 0.8% higher macro-F1 on MIMIC-III-full, p < 0.01), indicating its effectiveness in handling rare diagnoses.

Limitations and Future Work

  • Limitations:
    • The paper notes that on the MIMIC-III-50 dataset, the performance difference between LAAT and JointLAAT was not significant. This is attributed to the dataset having fewer infrequent codes and a less pronounced hierarchical structure among the top 50 codes.
  • While LAAT achieved slightly lower micro-AUC than MSATT-KG on MIMIC-III-full, the overall performance gains in other metrics suggest a trade-off that favors broader accuracy.
  • Future Work:
    • The authors suggest that the hierarchical joint learning mechanism is a promising direction for improving performance on infrequent codes, implying further exploration of different hierarchical structures or integration methods could be beneficial.
  • The effectiveness of BiLSTM over BiGRU for long text suggests that exploring other recurrent or attention-based architectures capable of handling long sequences could be a fruitful avenue.

Summary updated

BibTeX
@inproceedings{vu2020label,
  author = {Thanh Vu and Dat Quoc Nguyen and Anthony N. Nguyen},
  title = {{A Label Attention Model for ICD Coding from Clinical Text}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/461},
}
Figure