PLM-ICD: Automatic ICD Coding with Pretrained Language Models

Chao-Wei Huang, Shang-Chi Tsai, Yun-Nung Chen

TL;DR

This paper introduces PLM-ICD, a novel framework that leverages Pretrained Language Models (PLMs) for automatic ICD coding from clinical notes. By addressing key challenges like long input sequences, large label spaces, and domain mismatch, PLM-ICD achieves state-of-the-art performance on benchmark datasets, demonstrating the potential of PLMs in this critical healthcare NLP task.

What Problem Does This Paper Address?

Automatically assigning International Classification of Diseases (ICD) codes to clinical notes is a crucial but challenging task in healthcare. These codes are vital for tracking statistics, quality outcomes, and billing. However, manual coding is labor-intensive and requires specialized expertise. While NLP has made strides, existing methods often struggle with the unique characteristics of clinical notes:

  • Large Label Space: ICD coding involves classifying notes into one of thousands of possible diagnostic codes.
  • Long Input Sequences: Clinical notes can be lengthy, often exceeding the maximum input length of standard PLMs.
  • Domain Mismatch: General-domain PLMs, pre-trained on broad text corpora, may not effectively capture the nuances of medical language found in clinical notes.

Previous attempts to apply PLMs to this task have shown suboptimal performance compared to specialized models, highlighting the need for a tailored approach.

Methods

The authors propose PLM-ICD, a framework designed to overcome the aforementioned challenges:

  • Domain-Specific Pretraining: To address the domain mismatch, PLM-ICD suggests using PLMs that have already been pre-trained on biomedical and clinical text (e.g., BioBERT, PubMedBERT). This allows the model to better understand medical terminology from the outset.
  • Segment Pooling: To handle long input sequences, PLM-ICD splits the clinical note into smaller segments, each within the PLM's maximum length. These segments are then encoded individually, and their representations are aggregated to form a document-level representation. This is an improvement over simple truncation, which loses information.
  • Label-Aware Attention: To effectively manage the large label space, PLM-ICD incorporates a label-aware attention mechanism. This mechanism allows the model to learn label-specific representations by focusing on text fragments most relevant to each individual ICD code, rather than relying solely on a single [CLS] token representation.

Datasets

The proposed framework, PLM-ICD, was evaluated on two widely used benchmark datasets for ICD code prediction:

  • MIMIC-3: A large dataset containing clinical notes and structured data from intensive care units. The authors used 47,724 discharge summaries for training.
  • MIMIC-2: Another benchmark dataset used for comparison with prior work, with 20,533 summaries for training.

Key Insights and Ideas

  • PLMs are not plug-and-play for ICD coding: Simply fine-tuning general-domain PLMs with a standard linear layer is insufficient due to the inherent complexities of the task.
  • Domain-specific pretraining is crucial: Models pre-trained on biomedical text (like RoBERTa-PM) significantly outperform those trained on general text, underscoring the importance of domain adaptation.
  • The [CLS] token is not enough for multi-label classification: Relying solely on the [CLS] token's representation for large-scale multi-label tasks like ICD coding leads to inferior performance compared to methods that consider token-level information.
  • Label attention is key: The label-aware attention mechanism is vital for achieving high performance, as it allows the model to learn discriminative features for each specific ICD code.

Key Results

PLM-ICD achieved state-of-the-art performance on both MIMIC-3 and MIMIC-2 datasets across multiple metrics, including micro-F1 and precision@k. For instance, on MIMIC-3, PLM-ICD achieved a micro-F1 score of 59.8%, outperforming previous models. The authors also demonstrated that their segment pooling strategy was more effective than alternative methods for handling long input texts. Ablation studies confirmed the significant contributions of domain-specific pretraining, segment pooling, and label-aware attention to the overall performance.

Limitations and Future Work

The authors acknowledge that the fine-tuning process is sensitive to hyperparameters, and suggest that exploring different optimization schedules and learning rates can further improve performance. They also note that while their segment pooling method is effective, exploring PLMs with longer input length capabilities or recurrent mechanisms could be beneficial. Future work could also involve integrating external knowledge sources or exploring more advanced attention mechanisms. The paper also highlights that techniques like those used in RAC (which incorporate special code description modeling) are complementary and could potentially be integrated into PLM-ICD for further gains.

Summary updated

Abstract

Automatically classifying electronic health records (EHRs) into diagnostic codes has been challenging to the NLP community. State-of-the-art methods treated this problem as a multilabel classification problem and proposed various architectures to model this problem. However, these systems did not leverage the superb performance of pretrained language models, which achieved superb performance on natural language understanding tasks. Prior work has shown that pretrained language models underperformed on this task with the regular finetuning scheme. Therefore, this paper aims at analyzing the causes of the underperformance and developing a framework for automatic ICD coding with pretrained language models. We spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. We propose PLMICD, a framework that tackles the challenges with various strategies. The experimental results show that our proposed framework can overcome the challenges and achieves state-of-the-art performance in terms of multiple metrics on the benchmark MIMIC data. The source code is available at https://github.com/MiuLab/PLM-ICD
BibTeX
@inproceedings{huang2022plm,
  author = {Chao-Wei Huang and Shang-Chi Tsai and Yun-Nung Chen},
  title = {{PLM-ICD: Automatic ICD Coding with Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure