TL;DR
This paper introduces a "curriculum batching strategy" for training deep learning models to automatically assign ICD codes to medical records. By presenting data in a structured, easy-to-hard sequence, their method significantly improves accuracy, reduces overfitting, and enhances the model's ability to handle rare medical conditions, outperforming existing approaches.
What Problem Does This Paper Address?
Automatic ICD coding is crucial for organizing medical information, but it's a challenging task. The primary hurdles are the sheer number of possible ICD codes (labels) and the highly imbalanced distribution of these labels, meaning some codes are extremely common while others are very rare. When training deep learning models using standard minibatch gradient descent (MBGD), this imbalance causes the data distribution within each small batch to differ significantly from the overall training data distribution. This inconsistency can lead to models overfitting to common labels and performing poorly on rare ones, ultimately hurting their generalization ability.
Methods
The core innovation is a Curriculum Batching Strategy (CBS) that replaces the standard random shuffling of data into batches. Instead of random batches, CBS generates three distinct types of batch sets offline:
- Uniform Data Distribution (UDD): Achieved through Stratified Sampling with Replacement (SSR). This ensures that examples for each ICD code have an equal chance of being selected, helping to address the imbalanced label problem. This is considered the "easiest" to learn from.
- Shuffling Data Distribution (SDD): This is the standard random shuffling approach, serving as a baseline.
- Imbalanced Training Data Distribution (IDD): Achieved through Probability Sampling with Replacement (PSR), aiming to mimic the original imbalanced data distribution. This is considered the "hardest" to learn from.
The training process then iterates through these batch sets sequentially: UDD first, then SDD, and finally BIDD. This gradual progression from uniform to imbalanced data is inspired by curriculum learning, where models learn simpler concepts before tackling more complex ones.
Datasets
The primary dataset used is the MIMIC-III dataset, specifically the third version. The researchers adapted electronic medical records (discharge summaries) and their corresponding ICD-9 codes to create training examples. After preprocessing (removing punctuation, numbers, stop words, tokenizing, and selecting top 10,000 words based on TF-IDF), they obtained a dataset with 55,177 records and 6918 unique ICD-9 codes. This dataset was split into training, validation, and testing sets in a 7:1:2 ratio.
Key Insights and Ideas
- The "Curriculum" Matters: Presenting data in a structured, easy-to-hard sequence (UDD -> SDD -> BIDD) is more effective than random batching for deep multi-label classification tasks like ICD coding.
- Batch Distribution Inconsistency is a Real Problem: The paper provides visual evidence (Figure 1) showing how random batching can lead to local batch distributions that deviate significantly from the global training data distribution, contributing to overfitting.
- UDD is Key for Rare Labels: The UDD batch set, generated via stratified sampling, is crucial for ensuring that models get sufficient exposure to rare ICD codes, thereby improving their ability to learn the "long-tailed" label distribution.
- Faster Convergence and Better Generalization: The CBS not only improves final performance but also leads to faster convergence and better generalization, as evidenced by comparable performance on training and test sets.
Key Results
- Dramatic Performance Improvement: Replacing the standard shuffling algorithm with CBS led to more than double the F1micro scores for TextCNN, TextRNN, and TextRCNN models.
- Significant F1macro Gains: The F1macro score, which is more sensitive to long-tailed labels, increased from less than 5% to around 50%.
- Outperforming SOTA: The CBS-enhanced models outperformed a state-of-the-art label set reconstruction (ReLS) method.
- Reduced Overfitting: Models trained with CBS showed comparable performance on training and test data, indicating a significant reduction in overfitting.
- Improved Long-Tailed Label Learning: The paper demonstrates improved precision and recall for rare (long-tailed) labels when using CBS.
Limitations and Future Work
- Fixed Curriculum Order: The current CBS uses a fixed order (UDD, SDD, BIDD) and a fixed number of epochs for each batch type. The authors suggest that adaptively choosing the course of the CBS based on model convergence during training could be a future direction.
- Transfer Learning: Combining CBS with transfer learning and pre-training models on medical domain-specific datasets is proposed as another avenue for future research.
Summary updated