Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings

Goncalo Gomes, Isabel Coutinho, Bruno MartinsEACL 2024

TL;DR

This paper introduces a novel deep learning approach for automatically assigning International Classification of Diseases (ICD) codes to clinical text. By combining advanced text encoding with a unique label embedding mechanism that leverages ICD code synonyms, the model achieves state-of-the-art performance and produces well-calibrated predictions, which are crucial for downstream tasks like prevalence estimation.

What Problem Does This Paper Address?

Manually assigning ICD codes to clinical notes is a laborious, error-prone, and costly process. While automated methods using deep learning have shown promise, they face challenges due to the long, complex nature of clinical narratives, the specialized medical vocabulary, and the high-dimensional, imbalanced label space of ICD codes. Furthermore, accurately estimating the prevalence of ICD codes within a dataset (text quantification) is also important, and this requires well-calibrated classification models. This paper aims to address these issues by developing a more accurate and reliable automated ICD coding system.

Methods

The core of the proposed approach lies in its sophisticated text encoding and label embedding strategies.

  • Text Encoding: The authors explore two main strategies for handling long clinical documents using the GatorTron-base Transformer model:
  • Longformer Encoding (LE): Adapting the Transformer's self-attention mechanism to a sparse, windowed approach (Longformer) to efficiently process long sequences.
  • Chunk Encoding (CE): Dividing long documents into smaller chunks and processing each chunk independently with the standard Transformer encoder. This approach is favored for its ability to avoid issues with LMs struggling with very long inputs. They also employ a smooth partitioning scheme with overlaps between chunks to prevent information loss.
  • Multi-Synonym Attention (MSAM): To improve code representation, the model leverages a mechanism inspired by previous work that incorporates diverse synonyms of ICD codes. They gather synonyms from medical knowledge bases and even Wikipedia, then use a Maximum Diversity Problem (MDP) approach to select a representative subset of synonyms for each ICD code. This allows the model to attend to relevant parts of the text that are semantically related to the ICD code's synonyms, not just its primary description.
  • Joint Classification and Quantification (CLQ): To ensure well-calibrated predictions, the authors integrate a text quantification module (an MLP) with the classification model. This module is trained jointly with the classifier, optimizing a combined loss function that includes both classification loss (Binary Cross-Entropy) and quantification loss (MSE or Huber loss). This joint training aims to improve the model's ability to accurately estimate class prevalence and enhance overall calibration.

Datasets

The experiments were conducted on the publicly available MIMIC-III dataset. Specifically, two splits were used:

  • MIMIC-III-50: This split includes the top-50 most frequent ICD-9 codes.
  • MIMIC-III-clean: This is a cleaned version of the dataset containing 3,681 unique ICD-9 codes, addressing potential issues with previous data splits.

The authors note that these splits are standard in the field and were used to ensure comparability with prior research.

Key Insights and Ideas

  • Synonyms are Key: Leveraging diverse synonyms for ICD codes significantly enhances both classification accuracy and model calibration. This suggests that a richer understanding of code semantics, beyond just the primary description, is crucial for accurate coding.
  • Chunking Outperforms Global Attention for Long Texts: The Chunk Encoding (CE) strategy, which processes documents in segments, generally outperforms the Longformer Encoding (LE) approach that compresses the entire document into a single representation. This highlights the challenges Transformers face with extremely long contexts and the benefits of a more localized processing strategy.
  • Joint Training for Calibration: Training the classification and quantification objectives together leads to better-calibrated predictions. This is particularly evident when using the Huber loss for quantification, which appears to balance classification performance and calibration more effectively than MSE loss.
  • Few-Shot Learning Capabilities: The model demonstrates remarkable performance even on rare ICD codes and specific chapters with low prevalence, suggesting an ability to learn effectively from limited data.

Key Results

  • State-of-the-Art Performance: The proposed CE+MSAM+CLQHuber model achieved state-of-the-art results on both MIMIC-III-50 and MIMIC-III-clean datasets across various metrics, including micro-averaged F1 scores of 73.9% and 60.5%, respectively.
  • Improved Calibration: The label embedding mechanism and joint training significantly improved model calibration, as measured by the Mean Expected Calibration Error (MECE).
  • Quantification Accuracy: The joint training approach, especially with the Huber loss, demonstrated superior performance in ICD code quantification compared to baseline methods like CC, PCC, and a separately trained MLP.

Limitations and Future Work

  • Lack of Hierarchical Structure Exploitation: The current model does not explicitly leverage the inherent hierarchical structure of the ICD coding system. The authors suggest implementing dual classification heads as a future direction to incorporate this.
  • Alternative Calibration Methods: While joint training improved calibration, exploring other classification loss functions beyond BCE could further enhance this aspect, which is critical for accurate quantification.
  • Dataset Limitations: The experiments relied on existing MIMIC-III splits, which may not fully represent the vast diversity of clinical and linguistic data encountered in real-world deployments. Future work could involve testing on more diverse datasets.

Summary updated

BibTeX
@inproceedings{gomes2024accurate,
  author = {Goncalo Gomes and Isabel Coutinho and Bruno Martins},
  title = {{Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings}},
  booktitle = {EACL},
  year = {2024},
}
Figure