TL;DR
Automated ICD coding from free-text clinical discharge notes is notoriously difficult due to extreme document lengths, complex medical terminology, and vast label spaces. This paper introduces GatedCNN-NCI, an architecture that combines dilated convolutions, embedding injection, and depth-wise gating to encode long clinical notes, alongside a bipartite graph message-passing module to explicitly capture interactions between text and ICD code descriptions. The resulting model outperforms strong baselines across standard benchmarks while maintaining a parameter count comparable to lightweight models.
What Problem Does This Paper Address?
When patients are discharged from the hospital, clinical professionals assign standardized diagnosis and procedure codes (such as ICD-9 or ICD-10) based on free-text clinical notes. Automating this multi-label classification task is challenging for two major reasons:
- Lengthy and dense documentation: Discharge summaries easily span thousands of tokens filled with specialized clinical jargon, causing standard deep learning encoders to suffer from feature blurring or forgetting.
- High-dimensional, sparse label space: Coding systems contain tens of thousands of potential codes (e.g., >14,000 for ICD-9), yet individual patient stays typically require only a handful of specific codes.
Existing deep learning models often fail to capture long-range semantic dependencies in text or treat codes merely as isolated, discrete indices rather than leveraging the rich medical knowledge found in code descriptions.
Methods
GatedCNN-NCI introduces two core components: a specialized text encoder and a bipartite interaction module.
Clinical Note ──> [Word Embeddings]
│
▼
┌───────────────────────────────────┐
│ Stacked Gated Dilated CNNs │
│ • Embedding Injection: [X, H^l] │
│ • Depth-wise LSTM Gating │
│ • Parameter Weight Sharing │
└─────────────────┬─────────────────┘
│ Textual Features (U)
▼
[Bipartite Graph Interaction] <── ICD Code Descriptions (V)
│ Dot-product message passing (I = V U^T)
▼
[FCN + Sigmoid Output] ──> Code Predictions
- Stacked Gated Dilated Convolutions: Uses 1D dilated convolutions ($Conv1D$) to efficiently expand the receptive field across long token sequences without exploding computational cost.
- Embedding Injection: At each intermediate layer $l$, the initial word embedding matrix $\mathbf{X}$ is concatenated back into the hidden state $\mathbf{H}^l$. This prevents deep layers from "forgetting" critical diagnostic mentions.
- LSTM-Style Gating across Depth: Unlike standard LSTMs that gate sequentially along the time dimension, this mechanism regulates information propagation across network layers (depth-wise) using input, forget, output, and cell state gates.
- Note-Code Interaction (NCI) via Message Passing: Formulates clinical note tokens and ICD codes as a complete bipartite graph $\mathcal{G} = \{\mathcal{U}, \mathcal{V}, \mathcal{E}\}$. Label features are obtained by averaging word vectors from official ICD code text descriptions. A dot-product message-passing mechanism computes explicit token-code alignment scores before a fully connected network produces discharge code probabilities.
- Parameter-Efficient Weight Sharing: Weights of the gated CNN layers are tied across network depth and time steps, acting as a regularizer and drastically shrinking model size.
Datasets
The authors evaluated their model on two standard, publicly available electronic health record benchmarks:
- MIMIC-II: 22,815 discharge summaries, evaluated on the full ICD-9 code space using a standard 90/10 train-test split.
- MIMIC-III: Derived from the
noteeventstable (v1.4, containing 58,576 hospital admissions). Evaluated in two settings: - MIMIC-III Top-50: The 50 most frequent ICD-9 codes (train/dev/test: 8,066 / 1,573 / 1,729).
- MIMIC-III Full: The complete set of ICD-9 diagnosis and procedure codes.
No new datasets were introduced.
Key Insights and Ideas
- Gating across depth, not time: While recurrent gating across time steps is slow for long sequences, using gating mechanisms across convolutional layers filters out noise while preserving critical hierarchical features.
- Code descriptions matter: Rather than treating codes as orthogonal one-hot targets, utilizing the semantic representations of ICD descriptions enables point-wise mutual information modeling between symptoms and diagnoses.
- Bypassing the Transformer length bottleneck: While models like ClinicalBERT excel on general clinical NLP, standard transformers struggle with long context windows (e.g., restricted to 512 tokens). Truncating long discharge notes discards vital diagnostic clues; dilated CNNs with embedding injection handle full note lengths far more effectively.
Key Results
- MIMIC-III Top-50: GatedCNN-NCI set new state-of-the-art results across all metrics, achieving 62.9% Macro F1 (a 2.0% improvement over HyperCore) and 68.6% Micro F1 (a 1.6% improvement).
- MIMIC-III Full Codes: Outperformed baselines on Macro F1 (9.2% vs. 9.0% for HyperCore) and Micro F1 (56.3% vs. 55.1% for HyperCore), while placing second in Macro AUC-ROC (92.2% vs. 93.0%).
- Comparison to Transformers: On sequence-truncated inputs (510 tokens), GatedCNN-NCI outperformed BERT-base and ClinicalBERT on AUC-ROC (83.7% Macro vs. 81.0% for ClinicalBERT) and Precision@5 (56.6% vs. 54.5%).
- Efficiency: With weight sharing, the model requires only 7.6M parameters, compared to 11.9M for MultiResCNN and 113.8M for ClinicalBERT.
- Ablation Findings: Removing the NCI module caused Macro F1 on MIMIC-III top-50 to fall from 62.9% to 61.4%, and removing the gating mechanism dropped it to 60.2%, confirming the value of both architectural choices.
Limitations and Future Work
- Extreme Label Sparsity: Although the model improved full-code Macro F1 compared to baselines, absolute Macro F1 scores remain low on full sets (e.g., 9.2% on MIMIC-III and 6.4% on MIMIC-II), reflecting persistent challenges with extremely rare disease codes.
- Simple Text Representation of Codes: Code vectors were derived by simply averaging word embeddings of ICD descriptions, which ignores higher-level syntax or formal hierarchical ontologies present in medical taxonomies.
- Future Directions: Incorporating more explicit knowledge graphs, hierarchical ontology structures, or exploring efficient linear/sparse long-context Transformer attention mechanisms.
Summary updated