Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs

Nasir Hayat, Hazem Lashen, Farah E. ShamoutarXiv 2021 · OpenReview

TL;DR

This paper introduces CXR-ML-GZSL, a novel deep learning framework for multi-label classification of chest X-rays that can identify both known and previously unseen diseases. By jointly learning visual representations from X-ray images and semantic representations from medical text, the model aims to overcome the limitations of traditional supervised learning, which requires extensive labeled data for every possible disease.

What Problem Does This Paper Address?

Supervised deep learning models have achieved remarkable success in medical image analysis, including chest X-ray (CXR) diagnosis. However, a significant limitation is their inability to recognize diseases not encountered during training. This necessitates costly and time-consuming data collection and annotation for every new disease, making it impractical to build models that can diagnose all possible conditions, especially rare ones or those from novel outbreaks. Radiologists, on the other hand, can infer unseen diseases by leveraging learned attributes. This paper tackles the challenge of enabling AI models to mimic this ability by developing a generalized zero-shot learning approach for multi-label CXR classification.

Methods

The core of the proposed CXR-ML-GZSL framework is an end-to-end trainable network that learns to align visual and semantic representations in a shared latent space. Key components include:

  • Visual Encoder: A standard deep learning model (e.g., DenseNet-121) that extracts visual features from CXR images. This encoder is fine-tuned during training.
  • Semantic Encoder: Utilizes BioBERT, a pre-trained biomedical language model, to generate semantic embeddings for disease classes from medical text. This encoder is kept fixed.
  • Mapping Modules: Two modules, one for visual features and one for semantic embeddings, that project their respective representations into a common latent space.
  • Novel Learning Objective: A multi-component loss function designed to:
  • Rank Relevance: Ensure that relevant labels for an image are ranked higher than irrelevant ones (using a margin-based ranking loss, $L_{rank}$).
  • Align Modalities: Align the visual representations with their corresponding semantic representations in the latent space ($L_{align}$).
  • Preserve Inter-Class Consistency: Maintain the original semantic relationships between classes in the projected space ($L_{con}$).

This approach allows the model to learn visual representations guided by semantics, without requiring pre-trained visual feature extractors on external datasets like ImageNet.

Datasets

  • NIH Chest X-ray Dataset: This is the primary dataset used for training and evaluation. It contains 112,120 frontal X-ray images from over 30,000 patients, with each image labeled with up to 14 possible disease classes.
  • The dataset was split into training (70%), validation (10%), and testing (20%) sets.
  • For the experiments, 10 classes were designated as "seen" (trained on) and 4 as "unseen" (evaluated on, but not used during training). The seen classes included conditions like Atelectasis and Pneumothorax, while unseen classes included Edema and Pneumonia.
  • Crucially, images associated with any unseen class were excluded from the training set to adhere to the inductive ZSL setting.

Key Insights and Ideas

  • Bridging the Modality Gap: The paper highlights the power of combining visual information from medical images with semantic knowledge derived from rich medical literature (via BioBERT) to improve diagnostic capabilities.
  • End-to-End Learning for Visual Features: Unlike many prior works that rely on pre-trained, fixed visual encoders, this approach demonstrates the benefit of jointly learning and fine-tuning the visual encoder within the ZSL framework. This allows the visual representation to be specifically tailored for the task of aligning with semantic embeddings.
  • Multi-Label Generalization is Crucial for Medical Imaging: The inherent nature of medical conditions often means multiple diseases can co-exist in a single image. The paper's focus on multi-label generalized zero-shot learning is therefore highly relevant for real-world clinical applications.
  • BioBERT's Efficacy: The use of BioBERT for semantic embeddings is shown to be effective, leveraging its understanding of biomedical terminology for better class representation.

Key Results

  • State-of-the-Art Performance: The proposed CXR-ML-GZSL model (referred to as "OUReze" in the tables) significantly outperforms two strong multi-label generalized zero-shot learning baselines (LESA and MLZSL) on the NIH Chest X-ray dataset.
  • Improved AUROC: The model achieved the highest AUROC for unseen classes (0.66) and a harmonic mean AUROC of 0.72 across both seen and unseen classes.
  • Precision Gains: For instance, precision@2 saw a substantial improvement of 73.68% compared to the MLZSL baseline.
  • Class-wise Performance: Table 2 shows that CXR-ML-GZSL achieved the best AUROC for most seen classes and significantly outperformed baselines on unseen classes.
  • Qualitative Results: Figure 2 demonstrates the model's ability to correctly predict multiple seen and unseen labels, even for images with a high number of ground-truth labels.
  • Ablation Studies: The results indicate that incorporating both the alignment loss ($L_{align}$) and the semantic consistency regularizer ($L_{con}$) is crucial for improving performance on unseen classes, while performance on seen classes remains robust. End-to-end training also yielded better results than using offline visual encoders.

Limitations and Future Work

  • Dataset Size and Class Diversity: The authors acknowledge that the NIH dataset, while large, has a relatively limited number of classes compared to natural image benchmarks. They emphasize the need for more challenging benchmark datasets with greater class diversity for future work.
  • Random Seen/Unseen Split: The selection of seen and unseen classes was random. Future work could explore different splits and evaluate the method on other medical imaging tasks and institutions to assess generalizability.
  • Hyperparameter Tuning: While learning rate and gamma parameters were tuned, further improvements might be possible by tuning other hyperparameters like batch size and mapping module dimensions.
  • Clinical Deployment: For practical clinical use, the authors suggest that practitioners would be shown a ranked list of predicted diseases and could use sensitivity/specificity analysis to convert scores into binary predictions.

Summary updated

Abstract

Despite the success of deep neural networks in chest X-ray (CXR) diagnosis, supervised learning only allows the prediction of disease classes that were seen during training. At inference, these networks cannot predict an unseen disease class. Incorporating a new class requires the collection of labeled data, which is not a trivial task, especially for less frequently-occurring diseases. As a result, it becomes inconceivable to build a model that can diagnose all possible disease classes. Here, we propose a multi-label generalized zero shot learning (CXR-ML-GZSL) network that can simultaneously predict multiple seen and unseen diseases in CXR images. Given an input image, CXR-ML-GZSL learns a visual representation guided by the input's corresponding semantics extracted from a rich medical text corpus. Towards this ambitious goal, we propose to map both visual and semantic modalities to a latent feature space using a novel learning objective. The objective ensures that (i) the most relevant labels for the query image are ranked higher than irrelevant labels, (ii) the network learns a visual representation that is aligned with its semantics in the latent feature space, and (iii) the mapped semantics preserve their original inter-class representation. The network is end-to-end trainable and requires no independent pre-training for the offline feature extractor. Experiments on the NIH Chest X-ray dataset show that our network outperforms two strong baselines in terms of recall, precision, f1 score, and area under the receiver operating characteristic curve. Our code is publicly available at: https://github.com/nyuad-cai/CXR-ML-GZSL.git
BibTeX
@misc{hayat2021multi,
  author = {Nasir Hayat and Hazem Lashen and Farah E. Shamout},
  title = {{Multi-Label Generalized Zero Shot Learning for the Classification of Disease in Chest Radiographs}},
  year = {2021},
  eprint = {2107.06563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.06563},
}
Figure