Medical Code Assignment with Gated Convolution and Note-Code Interaction

Shaoxiong Ji, Shirui Pan, Pekka Marttinen

TL;DR

Automated ICD coding from free-text clinical discharge notes is notoriously difficult due to extreme document lengths, complex medical terminology, and vast label spaces. This paper introduces GatedCNN-NCI, an architecture that combines dilated convolutions, embedding injection, and depth-wise gating to encode long clinical notes, alongside a bipartite graph message-passing module to explicitly capture interactions between text and ICD code descriptions. The resulting model outperforms strong baselines across standard benchmarks while maintaining a parameter count comparable to lightweight models.

What Problem Does This Paper Address?

When patients are discharged from the hospital, clinical professionals assign standardized diagnosis and procedure codes (such as ICD-9 or ICD-10) based on free-text clinical notes. Automating this multi-label classification task is challenging for two major reasons:

  1. Lengthy and dense documentation: Discharge summaries easily span thousands of tokens filled with specialized clinical jargon, causing standard deep learning encoders to suffer from feature blurring or forgetting.
  2. High-dimensional, sparse label space: Coding systems contain tens of thousands of potential codes (e.g., >14,000 for ICD-9), yet individual patient stays typically require only a handful of specific codes.

Existing deep learning models often fail to capture long-range semantic dependencies in text or treat codes merely as isolated, discrete indices rather than leveraging the rich medical knowledge found in code descriptions.

Methods

GatedCNN-NCI introduces two core components: a specialized text encoder and a bipartite interaction module.

Clinical Note ──> [Word Embeddings]
                         │
                         ▼
        ┌───────────────────────────────────┐
        │ Stacked Gated Dilated CNNs        │
        │  • Embedding Injection: [X, H^l]  │
        │  • Depth-wise LSTM Gating         │
        │  • Parameter Weight Sharing       │
        └─────────────────┬─────────────────┘
                          │ Textual Features (U)
                          ▼
            [Bipartite Graph Interaction] <── ICD Code Descriptions (V)
                          │ Dot-product message passing (I = V U^T)
                          ▼
                 [FCN + Sigmoid Output] ──> Code Predictions
  • Stacked Gated Dilated Convolutions: Uses 1D dilated convolutions ($Conv1D$) to efficiently expand the receptive field across long token sequences without exploding computational cost.
  • Embedding Injection: At each intermediate layer $l$, the initial word embedding matrix $\mathbf{X}$ is concatenated back into the hidden state $\mathbf{H}^l$. This prevents deep layers from "forgetting" critical diagnostic mentions.
  • LSTM-Style Gating across Depth: Unlike standard LSTMs that gate sequentially along the time dimension, this mechanism regulates information propagation across network layers (depth-wise) using input, forget, output, and cell state gates.
  • Note-Code Interaction (NCI) via Message Passing: Formulates clinical note tokens and ICD codes as a complete bipartite graph $\mathcal{G} = \{\mathcal{U}, \mathcal{V}, \mathcal{E}\}$. Label features are obtained by averaging word vectors from official ICD code text descriptions. A dot-product message-passing mechanism computes explicit token-code alignment scores before a fully connected network produces discharge code probabilities.
  • Parameter-Efficient Weight Sharing: Weights of the gated CNN layers are tied across network depth and time steps, acting as a regularizer and drastically shrinking model size.

Datasets

The authors evaluated their model on two standard, publicly available electronic health record benchmarks:

  • MIMIC-II: 22,815 discharge summaries, evaluated on the full ICD-9 code space using a standard 90/10 train-test split.
  • MIMIC-III: Derived from the noteevents table (v1.4, containing 58,576 hospital admissions). Evaluated in two settings:
  • MIMIC-III Top-50: The 50 most frequent ICD-9 codes (train/dev/test: 8,066 / 1,573 / 1,729).
  • MIMIC-III Full: The complete set of ICD-9 diagnosis and procedure codes.

No new datasets were introduced.

Key Insights and Ideas

  • Gating across depth, not time: While recurrent gating across time steps is slow for long sequences, using gating mechanisms across convolutional layers filters out noise while preserving critical hierarchical features.
  • Code descriptions matter: Rather than treating codes as orthogonal one-hot targets, utilizing the semantic representations of ICD descriptions enables point-wise mutual information modeling between symptoms and diagnoses.
  • Bypassing the Transformer length bottleneck: While models like ClinicalBERT excel on general clinical NLP, standard transformers struggle with long context windows (e.g., restricted to 512 tokens). Truncating long discharge notes discards vital diagnostic clues; dilated CNNs with embedding injection handle full note lengths far more effectively.

Key Results

  • MIMIC-III Top-50: GatedCNN-NCI set new state-of-the-art results across all metrics, achieving 62.9% Macro F1 (a 2.0% improvement over HyperCore) and 68.6% Micro F1 (a 1.6% improvement).
  • MIMIC-III Full Codes: Outperformed baselines on Macro F1 (9.2% vs. 9.0% for HyperCore) and Micro F1 (56.3% vs. 55.1% for HyperCore), while placing second in Macro AUC-ROC (92.2% vs. 93.0%).
  • Comparison to Transformers: On sequence-truncated inputs (510 tokens), GatedCNN-NCI outperformed BERT-base and ClinicalBERT on AUC-ROC (83.7% Macro vs. 81.0% for ClinicalBERT) and Precision@5 (56.6% vs. 54.5%).
  • Efficiency: With weight sharing, the model requires only 7.6M parameters, compared to 11.9M for MultiResCNN and 113.8M for ClinicalBERT.
  • Ablation Findings: Removing the NCI module caused Macro F1 on MIMIC-III top-50 to fall from 62.9% to 61.4%, and removing the gating mechanism dropped it to 60.2%, confirming the value of both architectural choices.

Limitations and Future Work

  • Extreme Label Sparsity: Although the model improved full-code Macro F1 compared to baselines, absolute Macro F1 scores remain low on full sets (e.g., 9.2% on MIMIC-III and 6.4% on MIMIC-II), reflecting persistent challenges with extremely rare disease codes.
  • Simple Text Representation of Codes: Code vectors were derived by simply averaging word embeddings of ICD descriptions, which ignores higher-level syntax or formal hierarchical ontologies present in medical taxonomies.
  • Future Directions: Incorporating more explicit knowledge graphs, hierarchical ontology structures, or exploring efficient linear/sparse long-context Transformer attention mechanisms.

Summary updated

Abstract

Medical code assignment from clinical text is a fundamental task in clinical information system management. As medical notes are typically lengthy and the medical coding system's code space is large, this task is a long-standing challenge. Recent work applies deep neural network models to encode the medical notes and assign medical codes to clinical documents. However, these methods are still ineffective as they do not fully encode and capture the lengthy and rich semantic information of medical notes nor explicitly exploit the interactions between the notes and codes. We propose a novel method, gated convolutional neural networks, and a note-code interaction (GatedCNN-NCI), for automatic medical code assignment to overcome these challenges. Our methods capture the rich semantic information of the lengthy clinical text for better representation by utilizing embedding injection and gated information propagation in the medical note encoding module. With a novel note-code interaction design and a graph message passing mechanism, we explicitly capture the underlying dependency between notes and codes, enabling effective code prediction. A weight sharing scheme is further designed to decrease the number of trainable parameters. Empirical experiments on real-world clinical datasets show that our proposed model outperforms state-of-the-art models in most cases, and our model size is on par with light-weighted baselines.
BibTeX
@inproceedings{ji2021medical,
  author = {Shaoxiong Ji and Shirui Pan and Pekka Marttinen},
  title = {{Medical Code Assignment with Gated Convolution and Note-Code Interaction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=3nRAAjPVkE},
}

Loading related content...

Figure