SwinCheX: Multi-label classification on chest X-ray images with transformers

Sina Taslimi, Soroush Taslimi, Nima Fathi, Mohammadreza Salehi, Mohammad Hossein RohbanarXiv 2022 · DOI · arXiv

Abstract

Automated diagnosis of Chest X-rays (CXR) is a critical goal in medical imaging. SwinCheX introduces a multi-label classification model that replaces traditional Convolutional Neural Network (CNN) backbones with a hierarchical Swin Transformer. By leveraging a 3-layer Multi-Layer Perceptron (MLP) head and adhering to a rigorous patient-wise evaluation protocol on the massive ChestX-ray14 dataset, the model achieves a new state-of-the-art (SOTA) average AUC of 0.810.

Introduction

Automated CXR diagnosis is essential for fast, reliable screening. While CNNs like ResNet and DenseNet have been standard, they often struggle with long-range dependencies and multi-scale features. Furthermore, the field faces a "fairness" crisis due to inconsistent data splits where images from the same patient appear in both training and testing sets (data leakage), leading to inflated performance metrics. SwinCheX addresses these issues with a more powerful architecture and a standardized benchmarking process using the official patient-wise split of the ChestX-ray14 dataset, which contains 112,120 images from over 30,000 patients across 14 pathologies.

Methods

The SwinCheX architecture departs from the CNN-heavy status quo by using a Swin Transformer (Swin-L) as the backbone.

The SwinCheX architecture featuring a Swin Transformer backbone and multi-task MLP heads.

Architectural Components

  • Swin Transformer Backbone: Unlike the original Vision Transformer (ViT), the Swin Transformer uses a hierarchical approach with "shifted windows." This captures multi-resolution features—essential for detecting both large-scale heart issues and tiny lung nodules—while maintaining linear complexity relative to image size. The backbone is pre-trained on ImageNet-22k and fine-tuned on medical data.
  • Multi-Task MLP Heads: The model employs 14 distinct MLP heads, one for each pathology. Experiments with various depths revealed that a 3-layer head (384, 48, and 48 neurons) provided the optimal balance of complexity and performance.

Results and Discussion

SwinCheX achieved a mean AUC of 0.810, surpassing the previous best of 0.799 when evaluated under the same fair, patient-wise protocol.

Key Insights

  • Hierarchical Features: The Swin Transformer significantly outperformed the standard ViT (0.810 vs 0.779 AUC), suggesting that hierarchical spatial representations are vital for medical imaging where pathology sizes vary.
  • Impact of the MLP Head: A "headless" Transformer using raw output tokens is insufficient. Adding a 3-layer MLP head allowed the model to learn more complex, disease-specific functions, boosting the mean AUC from 0.804 to 0.810.
Comparison of AUC performance across different MLP head configurations (headless to 3-layers).
  • Standardized Evaluation: Re-evaluating the previous SOTA (DNet) using strict patient-wise protocols showed a performance drop from 0.807 to 0.799, highlighting the impact of data leakage in prior research.
  • Explainability: Using Grad-CAM saliency maps, the authors confirmed the model focuses on correct anatomical regions, such as heart boundaries for Cardiomegaly, rather than image artifacts or shortcuts.

Conclusion

SwinCheX sets a new benchmark for CXR diagnosis by combining hierarchical transformers with robust evaluation standards. While successful, limitations include label noise in the ChestX-ray14 dataset (~90% accuracy) and image artifacts like watermarks. Future work should focus on specialized pre-processing to remove non-clinical features and optimizing computational costs for resource-constrained clinical settings.

Summary updated

Abstract

According to the considerable growth in the avail of chest X-ray images in diagnosing various diseases, as well as gathering extensive datasets, having an automated diagnosis procedure using deep neural networks has occupied the minds of experts. Most of the available methods in computer vision use a CNN backbone to acquire high accuracy on the classification problems. Nevertheless, recent researches show that transformers, established as the de facto method in NLP, can also outperform many CNN-based models in vision. This paper proposes a multi-label classification deep model based on the Swin Transformer as the backbone to achieve state-of-the-art diagnosis classification. It leverages Multi-Layer Perceptron, also known as MLP, for the head architecture. We evaluate our model on one of the most widely-used and largest x-ray datasets called "Chest X-ray14," which comprises more than 100,000 frontal/back-view images from over 30,000 patients with 14 famous chest diseases. Our model has been tested with several number of MLP layers for the head setting, each achieves a competitive AUC score on all classes. Comprehensive experiments on Chest X-ray14 have shown that a 3-layer head attains state-of-the-art performance with an average AUC score of 0.810, compared to the former SOTA average AUC of 0.799. We propose an experimental setup for the fair benchmarking of existing methods, which could be used as a basis for the future studies. Finally, we followed up our results by confirming that the proposed method attends to the pathologically relevant areas of the chest.
BibTeX
@misc{taslimi2022swinchex,
  author = {Sina Taslimi and Soroush Taslimi and Nima Fathi and Mohammadreza Salehi and Mohammad Hossein Rohban},
  title = {{SwinCheX: Multi-label classification on chest X-ray images with transformers}},
  year = {2022},
  eprint = {2206.04246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.04246},
}
Figure