Leveraging Hierarchical Category Knowledge for Data-Imbalanced Multi-Label Diagnostic Text Understanding

Shang-Chi Tsai, Ting-Yun Chang, Yun-Nung ChenLOUHI @ EMNLP 2019 · OpenReview

TL;DR

This paper tackles the challenge of accurately predicting medical diagnostic codes from clinical notes, which are often imbalanced and numerous. They introduce novel methods to leverage the inherent hierarchical structure of these codes, significantly improving prediction accuracy, especially for less frequent diagnoses.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) contain valuable clinical notes that describe patient symptoms and conditions. These notes are typically accompanied by diagnostic codes, such as those from the International Classification of Diseases (ICD), which are crucial for billing, research, and understanding patient health trajectories. The task of automatically predicting these diagnostic codes from clinical text is complex due to several factors:

  • Vast Label Space: The ICD system has tens of thousands of codes (ICD-9 has over 15,000, ICD-10 over 140,000), making it a massive multi-label classification problem.
  • Data Imbalance: Many diagnostic codes are rarely used, while a few are very common. This imbalance makes it difficult for machine learning models to learn to predict the less frequent codes effectively.
  • Noisy Text: Clinical notes can contain abbreviations, misspellings, and irrelevant information, further complicating the task.

Previous approaches often treated each diagnostic code independently, failing to leverage the rich semantic relationships between them. This paper aims to bridge this gap by incorporating the hierarchical structure of ICD codes into the prediction models.

Methods

The core of this work lies in integrating hierarchical category knowledge into existing convolutional neural network (CNN) models for diagnostic code prediction. The authors propose several mechanisms to achieve this:

  • Convolutional Models (TextCNN and CAML): They build upon established CNN architectures. TextCNN uses convolutional and max-pooling operations to extract features from text. CAML (Convolutional Attention Model) enhances this by incorporating an attention mechanism that focuses on text segments most relevant to specific diagnostic codes, addressing the data imbalance issue by leveraging code definitions.
  • Knowledge Integration Mechanisms:
    • Cluster Penalty: This method introduces additional loss terms (Ωbetween and Ωwithin) to encourage the model's parameters for codes within the same category to be similar. This helps share information across related diagnoses.
  • Multi-Task Learning: The model is trained to predict both the low-level diagnostic codes and their corresponding high-level categories simultaneously. This allows the model to learn general concepts from the higher categories, which can benefit the prediction of less frequent low-level codes.
  • Hierarchical Learning: This approach explicitly maps low-level codes to their high-level categories. Two strategies are proposed for estimating high-level category probabilities:
  • Average Meta-Label: The probability of a high-level category is the average of the probabilities of its constituent low-level codes.
  • At-least-one Meta-Label: The probability is derived from whether at least one of the low-level codes within that category is predicted.

These mechanisms are integrated into the CNN models and trained end-to-end, with a hyperparameter (λ) controlling the influence of the high-level category loss.

Datasets

The proposed methods were evaluated on two datasets:

  • MIMIC-3: A widely used benchmark dataset containing clinical notes and associated ICD codes from intensive care units. The authors used both the full MIMIC-3 dataset and a subset called MIMIC3-50, which includes the top 50 most frequent labels.
  • NTUH Dataset: A newly collected dataset from an internal hospital, characterized by shorter clinical notes, a higher Out-of-Vocabulary (OOV) rate, and a highly imbalanced distribution of ICD-9 codes. This dataset is presented as more challenging than MIMIC-3.

Key Insights and Ideas

  • Hierarchical Structure is Key: The most significant insight is that explicitly leveraging the hierarchical relationships between diagnostic codes (e.g., grouping "schizophrenia" and "anorexia nervosa" under "mental disorders") provides crucial semantic cues that improve prediction accuracy.
  • Sharing is Caring (for Rare Codes): By forcing related codes to share parameters or learn from common higher-level categories, models can learn more effectively from limited data for rare diagnoses.
  • Multi-Task Learning Offers Flexibility: The multi-task learning approach, predicting both low-level and high-level codes, proved particularly effective on the more challenging MIMIC3-Full dataset, suggesting its robustness.
  • Average Meta-Label is Promising: The "average meta-label" strategy within hierarchical learning showed strong performance, indicating that averaging probabilities across related codes is a simple yet effective way to leverage hierarchical information.

Key Results

The proposed knowledge integration mechanisms consistently improved performance across various metrics (Precision@K, MAP, Macro/Micro F1, Macro/Micro AUC) on both MIMIC-3 datasets compared to baseline CNN and CAML models.

  • On MIMIC3-50, all proposed mechanisms showed improvement, with hierarchical learning (average meta-label) performing best.
  • On MIMIC3-Full, multi-task learning provided the best performance, outperforming the baseline and even improving upon the state-of-the-art CAML model.
  • The NTUH dataset, being more challenging, also showed improvements with the proposed methods, demonstrating their generalization capability.
  • Qualitatively, the proposed methods tended to predict more relevant labels and could correct misclassifications by identifying the correct category for a wrongly predicted code.

Limitations and Future Work

  • Limitations: The paper acknowledges that the proposed methods tend to predict more labels than baselines, which might lead to some false positives. The performance on the NTUH dataset was lower than MIMIC-3, highlighting the challenges posed by shorter texts and higher OOV rates.
  • Future Work: The authors plan to analyze the performance of each individual label to understand which ones benefit most from the proposed approaches. They also aim to explore further ways to integrate domain knowledge into current machine learning models.

Summary updated

BibTeX
@inproceedings{tsai2019leveraging,
  author = {Shang-Chi Tsai and Ting-Yun Chang and Yun-Nung Chen},
  title = {{Leveraging Hierarchical Category Knowledge for Data-Imbalanced Multi-Label Diagnostic Text Understanding}},
  booktitle = {LOUHI @ EMNLP},
  year = {2019},
  url = {https://openreview.net/forum?id=ZMuWiEPiVWM},
}
Figure