PLM-ICD: Automatic ICD Coding with Pretrained Language Models

Chao-Wei Huang, Shang-Chi Tsai, Yun-Nung Chen

TL;DR

This paper introduces PLM-ICD, a novel framework that leverages pretrained language models (PLMs) for automatic ICD coding from clinical notes. By addressing key challenges like long input sequences, large label spaces, and domain mismatch, PLM-ICD achieves state-of-the-art performance on the MIMIC-3 and MIMIC-2 benchmarks, demonstrating the power of adapted PLMs for this critical healthcare task.

What Problem Does This Paper Address?

Accurately assigning International Classification of Diseases (ICD) codes to clinical notes is crucial for healthcare statistics, quality assessment, and billing. However, this task is incredibly labor-intensive and requires specialized medical knowledge. While automated ICD coding has seen progress with deep learning, existing methods haven't fully capitalized on the remarkable capabilities of modern pretrained language models (PLMs) like BERT. Prior attempts to use PLMs for ICD coding have underperformed due to several inherent challenges: clinical notes are often much longer than the maximum input length PLMs can handle, the sheer number of possible ICD codes (labels) is vast, and general-domain PLMs struggle with the specialized medical language and jargon found in clinical text.

Methods

The authors propose PLM-ICD, a framework designed to overcome these challenges. It comprises three core components:

  • Domain-Specific Pretraining: To address the domain mismatch, PLM-ICD suggests using PLMs that have already been pretrained on biomedical and clinical text (e.g., BioBERT, PubMedBERT). This allows the model to better understand medical terminology and context.
  • Segment Pooling: To handle long clinical notes, the input text is split into smaller segments. Each segment is then processed by the PLM to generate representations. These segment representations are subsequently aggregated (pooled) to form a document-level representation, effectively bypassing the PLM's fixed input length limitation.
  • Label-Aware Attention: To manage the large number of ICD codes, a label-aware attention mechanism is employed. This mechanism allows the model to focus on specific parts of the clinical note that are most relevant to each individual ICD code, leading to more accurate predictions.

Datasets

The study utilizes two widely recognized benchmark datasets for ICD code prediction:

  • MIMIC-3: A large, publicly available dataset containing ICU records, including discharge summaries. The authors used 47,724 summaries for training, with separate validation and testing sets. It contains 8,922 ICD codes.
  • MIMIC-2: Another dataset used for comparison with prior work, with 20,533 summaries for training and 2,282 for testing, featuring 5,031 labels.

Key Insights and Ideas

  • PLMs are powerful but need adaptation: While PLMs excel at general NLP tasks, simply fine-tuning them with a standard linear layer isn't enough for complex, large-scale tasks like ICD coding. The specific challenges of the domain require tailored approaches.
  • Long text is a major hurdle: Truncating clinical notes to fit PLM input limits significantly degrades performance. A strategy like segment pooling is essential for capturing the full context.
  • Domain-specific pretraining matters: Using PLMs already familiar with medical jargon provides a substantial advantage over general-domain models.
  • Label attention is crucial for scale: For tasks with thousands of labels, a mechanism that allows the model to attend to label-specific information is vital for effective classification.

Key Results

PLM-ICD achieved state-of-the-art performance on the MIMIC-3 dataset, outperforming previous methods in micro F1 and precision@k metrics, even without specialized code description modeling. On MIMIC-2, it also demonstrated state-of-the-art results across multiple metrics. Ablation studies confirmed the effectiveness of each component: domain-specific pretraining, segment pooling, and label-aware attention all contributed significantly to the overall performance improvement.

Limitations and Future Work

The authors acknowledge that while their framework is effective, there's room for further improvement. They suggest exploring PLMs with even longer input handling capabilities or recurrent mechanisms. They also highlight that the [CLS] token representation alone might be insufficient for large-scale multi-label classification, reinforcing the value of label attention. The importance of pretraining corpora for domain-specific tasks is emphasized, and they note that optimization hyperparameters significantly impact performance, suggesting careful tuning is necessary. Future work could involve integrating techniques like those used in models like RAC, which complement their approach.

Summary updated

Abstract

Automatically classifying electronic health records (EHRs) into diagnostic codes has been challenging to the NLP community. State-of-the-art methods treated this problem as a multilabel classification problem and proposed various architectures to model this problem. However, these systems did not leverage the superb performance of pretrained language models, which achieved superb performance on natural language understanding tasks. Prior work has shown that pretrained language models underperformed on this task with the regular finetuning scheme. Therefore, this paper aims at analyzing the causes of the underperformance and developing a framework for automatic ICD coding with pretrained language models. We spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. We propose PLMICD, a framework that tackles the challenges with various strategies. The experimental results show that our proposed framework can overcome the challenges and achieves state-of-the-art performance in terms of multiple metrics on the benchmark MIMIC data. The source code is available at https://github.com/MiuLab/PLM-ICD
BibTeX
@inproceedings{huang2022plm,
  author = {Chao-Wei Huang and Shang-Chi Tsai and Yun-Nung Chen},
  title = {{PLM-ICD: Automatic ICD Coding with Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure