A Curriculum Batching Strategy for Automatic ICD Coding with Deep Multi-Label Classification Models

Yaqiang Wang, Xu Han, Xuechao Hao, Tao Zhu, Hongping ShuHealthcare 2022 · DOI

TL;DR

This paper introduces a "curriculum batching strategy" for training deep learning models to automatically assign ICD codes to medical records. By presenting data in a structured, easy-to-hard sequence, their method significantly improves accuracy, reduces overfitting, and enhances the model's ability to handle rare medical conditions, outperforming existing approaches.

What Problem Does This Paper Address?

Automatic ICD coding is crucial for organizing medical information, but it's a challenging task. The primary hurdles are the sheer number of possible ICD codes (labels) and the highly imbalanced distribution of these labels, meaning some codes are extremely common while others are very rare. When training deep learning models using standard minibatch gradient descent (MBGD), this imbalance causes the data distribution within each small batch to differ significantly from the overall training data distribution. This inconsistency can lead to models overfitting to common labels and performing poorly on rare ones, ultimately hurting their generalization ability.

Methods

The core innovation is a Curriculum Batching Strategy (CBS) that replaces the standard random shuffling of data into batches. Instead of random batches, CBS generates three distinct types of batch sets offline:

  1. Uniform Data Distribution (UDD): Achieved through Stratified Sampling with Replacement (SSR). This ensures that examples for each ICD code have an equal chance of being selected, helping to address the imbalanced label problem. This is considered the "easiest" to learn from.
  2. Shuffling Data Distribution (SDD): This is the standard random shuffling approach, serving as a baseline.
  3. Imbalanced Training Data Distribution (IDD): Achieved through Probability Sampling with Replacement (PSR), aiming to mimic the original imbalanced data distribution. This is considered the "hardest" to learn from.

The training process then iterates through these batch sets sequentially: UDD first, then SDD, and finally BIDD. This gradual progression from uniform to imbalanced data is inspired by curriculum learning, where models learn simpler concepts before tackling more complex ones.

Datasets

The primary dataset used is the MIMIC-III dataset, specifically the third version. The researchers adapted electronic medical records (discharge summaries) and their corresponding ICD-9 codes to create training examples. After preprocessing (removing punctuation, numbers, stop words, tokenizing, and selecting top 10,000 words based on TF-IDF), they obtained a dataset with 55,177 records and 6918 unique ICD-9 codes. This dataset was split into training, validation, and testing sets in a 7:1:2 ratio.

Key Insights and Ideas

  • The "Curriculum" Matters: Presenting data in a structured, easy-to-hard sequence (UDD -> SDD -> BIDD) is more effective than random batching for deep multi-label classification tasks like ICD coding.
  • Batch Distribution Inconsistency is a Real Problem: The paper provides visual evidence (Figure 1) showing how random batching can lead to local batch distributions that deviate significantly from the global training data distribution, contributing to overfitting.
  • UDD is Key for Rare Labels: The UDD batch set, generated via stratified sampling, is crucial for ensuring that models get sufficient exposure to rare ICD codes, thereby improving their ability to learn the "long-tailed" label distribution.
  • Faster Convergence and Better Generalization: The CBS not only improves final performance but also leads to faster convergence and better generalization, as evidenced by comparable performance on training and test sets.

Key Results

  • Dramatic Performance Improvement: Replacing the standard shuffling algorithm with CBS led to more than double the F1micro scores for TextCNN, TextRNN, and TextRCNN models.
  • Significant F1macro Gains: The F1macro score, which is more sensitive to long-tailed labels, increased from less than 5% to around 50%.
  • Outperforming SOTA: The CBS-enhanced models outperformed a state-of-the-art label set reconstruction (ReLS) method.
  • Reduced Overfitting: Models trained with CBS showed comparable performance on training and test data, indicating a significant reduction in overfitting.
  • Improved Long-Tailed Label Learning: The paper demonstrates improved precision and recall for rare (long-tailed) labels when using CBS.

Limitations and Future Work

  • Fixed Curriculum Order: The current CBS uses a fixed order (UDD, SDD, BIDD) and a fixed number of epochs for each batch type. The authors suggest that adaptively choosing the course of the CBS based on model convergence during training could be a future direction.
  • Transfer Learning: Combining CBS with transfer learning and pre-training models on medical domain-specific datasets is proposed as another avenue for future research.

Summary updated

Abstract

The International Classification of Diseases (ICD) has an important role in building applications for clinical medicine. Extremely large ICD coding label sets and imbalanced label distribution bring the problem of inconsistency between the local batch data distribution and the global training data distribution into the minibatch gradient descent (MBGD)-based training procedure for deep multi-label classification models for automatic ICD coding. The problem further leads to an overfitting issue. In order to improve the performance and generalization ability of the deep learning automatic ICD coding model, we proposed a simple and effective curriculum batching strategy in this paper for improving the MBGD-based training procedure. This strategy generates three batch sets offline through applying three predefined sampling algorithms. These batch sets satisfy a uniform data distribution, a shuffling data distribution and the original training data distribution, respectively, and the learning tasks corresponding to these batch sets range from simple to complex. Experiments show that, after replacing the original shuffling algorithm-based batching strategy with the proposed curriculum batching strategy, the performance of the three investigated deep multi-label classification models for automatic ICD coding all have dramatic improvements. At the same time, the models avoid the overfitting issue and all show better ability to learn the long-tailed label information. The performance is also better than a SOTA label set reconstruction model.
BibTeX
@inproceedings{wang2022curriculum,
  author = {Yaqiang Wang and Xu Han and Xuechao Hao and Tao Zhu and Hongping Shu},
  title = {{A Curriculum Batching Strategy for Automatic ICD Coding with Deep Multi-Label Classification Models}},
  booktitle = {Healthcare},
  year = {2022},
  doi = {10.3390/healthcare10122397},
}
Figure