HydraViT: Adaptive Multi-Branch Transformer for Multi-Label Disease Classification from Chest X-ray Images

Şaban Öztürk, Mehmet Y. Turali, Tolga ÇukurBiomedical Signal Processing and Control 2023

TL;DR

This paper introduces HydraViT, a novel hybrid convolutional-transformer model for multi-label chest X-ray (CXR) disease classification. By combining a spatial encoder for local features and a transformer for long-range context, along with an adaptive multi-branch output, HydraViT significantly improves classification accuracy and robustness, especially for co-occurring pathologies.

What Problem Does This Paper Address?

Diagnosing chest diseases from X-rays is crucial but challenging due to variations in pathology size, location, and visual similarity. Existing deep learning models, particularly CNNs, struggle with the "locality bias" and often fail to capture complex co-occurrence patterns between different diseases. While attention mechanisms and transformers have been used to improve focus and long-range context, effectively handling the heterogeneity of pathology and the intricate relationships between multiple diseases in a single CXR remains a significant hurdle for achieving robust and accurate multi-label classification.

Methods

HydraViT employs a three-part architecture:

  1. Spatial Encoder (SE): A CNN-based module (specifically, a pre-trained VGG16) that extracts local spatial features from CXR images and reduces their dimensionality. This step helps in capturing fine-grained details within the image.
  2. Context Encoder (CE): A transformer-based module that takes the feature maps from the SE and processes them to capture long-range contextual relationships. It splits the image into patches, embeds them, and uses multi-head self-attention to understand how different parts of the image relate to each other, crucial for identifying dispersed or overlapping pathologies.
  3. Multi-Branch Output (MBO): This is the core innovation. Instead of a single output layer, HydraViT uses a multi-branch approach. It has dedicated output branches for each individual disease label to ensure sensitivity to each specific condition. Crucially, it also includes an aggregated branch that considers all labels together to better capture co-occurrence relationships. To balance the importance of individual and aggregated predictions, HydraViT uses learnable weights for each branch and a consistency loss to ensure their outputs align.

Datasets

The primary dataset used for training and evaluation is the ChestX-ray14 dataset. This dataset contains 112,120 frontal-view CXR images from 30,805 patients, labeled with 15 disease classes (including 'No Finding'). The dataset is noted for its heterogeneity in pathology size (average ~7.5% of image size) and the frequent co-occurrence of multiple diseases. The authors preprocessed the images by downsampling them to 224x224 and split the data into training (86,524 images) and testing (25,596 images) sets with no patient overlap.

Key Insights and Ideas

  • The Power of Hybridization: Combining CNNs for local feature extraction with transformers for global context is a potent strategy for complex image analysis tasks like CXR interpretation.
  • Multi-Task Learning for Co-occurrence: The key insight is that a single output layer struggles with multi-label problems where diseases often appear together. HydraViT's multi-branch output, with separate heads for individual diseases and a combined head for co-occurrences, directly addresses this.
  • Adaptive Weighting is Crucial: Simply having multiple branches isn't enough. The learnable weights and consistency loss in HydraViT's MBO module are vital for adapting to the varying prevalence and co-occurrence statistics of different diseases, leading to more balanced performance.
  • Transformer's Advantage in Context: The transformer's ability to model long-range dependencies is particularly beneficial for CXR, where pathologies can be spread out or subtle, requiring a broader view than traditional CNNs might offer.

Key Results

HydraViT demonstrates superior performance compared to various state-of-the-art methods:

  • Overall Improvement: On average, HydraViT outperforms attention-guided methods by 1.2%, region-guided methods by 1.4%, and semantic-guided methods by 1.0% in AUC.
  • Ablation Studies:
    • Compared to a model with only the spatial encoder (fSE), HydraViT shows a 5.9% AUC improvement.
  • Adding the context encoder (fSE + CE) improved AUC by 4.8% over fSE.
  • Adding the multi-branch output (fSE + fMBO) improved AUC by 3.1% over fSE.
  • HydraViT consistently outperformed an aggregated variant (fSE + CE with a single softmax output) by 4.8% and an ensemble variant (multiple single-label models) by 9.4% AUC, highlighting the benefits of its integrated multi-task approach.
  • Homogeneity: HydraViT achieved the lowest standard deviation in AUC across labels, indicating more consistent performance across all disease types, including rarer ones.
  • Qualitative Results: Visualizations (GradCAM heatmaps) show that HydraViT effectively focuses on pathological regions and generates accurate predictions, with high confidence scores for ground-truth labels.

Limitations and Future Work

The authors acknowledge several areas for future work:

  • Computational Complexity: While HydraViT is efficient, a pure transformer architecture could offer further improvements at the cost of increased complexity. Low-rank approximations or pyramidal transformers could be explored.
  • Noisy Annotations: The CXR dataset's labels are derived from radiological reports and may contain noise. Future work could explore learning procedures that are more robust to such label noise.
  • Class Imbalance: While HydraViT mitigates some issues, class imbalance remains a challenge. Investigating methods that use embedding distances in latent space could further improve discrimination.
  • Context Encoder Enhancements: Exploring alternatives to self-attention, such as focal modulation networks, could potentially enhance sensitivity to co-occurrence relationships.

Summary updated

Abstract

Chest X-ray is an essential diagnostic tool in the identification of chest diseases given its high sensitivity to pathological abnormalities in the lungs. However, image-driven diagnosis is still challenging due to heterogeneity in size and location of pathology, as well as visual similarities and co-occurrence of separate pathology. Since disease-related regions often occupy a relatively small portion of diagnostic images, classification models based on traditional convolutional neural networks (CNNs) are adversely affected given their locality bias. While CNNs were previously augmented with attention maps or spatial masks to guide focus on potentially critical regions, learning localization guidance under heterogeneity in the spatial distribution of pathology is challenging. To improve multi-label classification performance, here we propose a novel method, HydraViT, that synergistically combines a transformer backbone with a multi-branch output module with learned weighting. The transformer backbone enhances sensitivity to long-range context in X-ray images, while using the self-attention mechanism to adaptively focus on task-critical regions. The multi-branch output module dedicates an independent branch to each disease label to attain robust learning across separate disease classes, along with an aggregated branch across labels to maintain sensitivity to co-occurrence relationships among pathology. Experiments demonstrate that, on average, HydraViT outperforms competing attention-guided methods by 1.2%, region-guided methods by 1.4%, and semantic-guided methods by 1.0% in multi-label classification performance.
BibTeX
@article{ozturk2023hydravit,
  author = {Şaban Öztürk and Mehmet Y. Turali and Tolga Çukur},
  title = {{HydraViT: Adaptive Multi-Branch Transformer for Multi-Label Disease Classification from Chest X-ray Images}},
  journal = {Biomedical Signal Processing and Control},
  year = {2023},
}
Figure