ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representations

Chinmay Prabhakar, Hongwei Bran Li, Jiancheng Yang, Suprosana Shit, Benedikt Wiestler, Bjoern Menze

Self-supervised learning (SSL) has become a cornerstone of modern computer vision, allowing models to learn rich representations from vast amounts of unlabeled data. In the medical domain, where expert annotations are expensive and scarce, SSL is particularly transformative. This paper introduces ViT-AE++, an evolution of the Vision Transformer Autoencoder (ViT-AE) that combines the best of reconstruction-based learning and contrastive learning to master both 2D and 3D medical image representations.

TL;DR

ViT-AE++ enhances the standard Vision Transformer Autoencoder by integrating auxiliary structural losses (perceptual and edge-aware) and a contrastive learning objective. This hybrid approach significantly improves representation quality for medical imaging, outperforming vanilla ViT-AE and standalone contrastive methods on both 2D X-rays and 3D MRI volumes.

What Problem Does This Paper Address?

While Vision Transformer Autoencoders (like MAE) have shown great promise by learning to reconstruct masked image patches, they suffer from two main drawbacks in the medical context:

  1. Pixel-wise Limitations: Standard models typically use a simple L2 (mean-squared error) loss for reconstruction. This ignores the "structured dependencies"—the complex textures and morphologies—that are vital for medical diagnosis.
  2. Indirect Learning: In a pure autoencoder, the latent representation is learned indirectly as a byproduct of reconstruction. There is no explicit objective to ensure that two different views of the same anatomy result in similar feature embeddings.

ViT-AE++ tackles these gaps by forcing the model to care about edges and high-level features while explicitly aligning representations through contrastive learning.

Methods

The authors propose a multi-faceted loss function to train a 3D Vision Transformer. The framework, illustrated in Figure 1, uses a "shared encoder" architecture to process two different augmented and masked versions of the same input.

Figure 1: Our proposed ViT-AE++ framework. The upper half shows the training procedure of the autoencoder. An MRI volume is parsed and masked randomly and the visible patches are passed through to the encoder. Placeholder tokens for the masked patches are introduced after the encoder (referred to as the MASK tokens). The full set of encoded features and these placeholder tokens are passed to a decoder which reconstructs the whole MRI volume. We use a pixel-wise reconstruction loss and an auxiliary loss that concerns perceptual features (Eq. 1) and edge map (Eq. 2). The lower half demonstrates the optimization with a contrastive loss which is based on two randomly augmented and masked views. The representations from the two views are matched with cosine similarity loss (Eq. 3).

The total loss is a combination of four terms:

  • Pixel-wise Reconstruction ($L_{rec}$): The standard L2 loss for patch recovery.
  • Perceptual Loss ($L_{per}$): Uses a pre-trained VGG network to compare high-level features of 2D slices, ensuring the model captures semantic structure.
  • Edge-aware Loss ($L_{edge}$): A novel 3D Sobel filter-based loss that penalizes discrepancies in gradients across axial, coronal, and sagittal planes.
  • Contrastive Loss ($L_{CL}$): Borrowing from the SimSiam paradigm, this term directly maximizes the similarity between the latent representations of two different masked views of the same image.

Datasets

The model was rigorously tested across natural and medical domains:

  • 2D Natural Images: CIFAR-10, CIFAR-100, and Tiny ImageNet-100.
  • 2D Medical Images: A Chest X-ray dataset for pneumonia identification.
  • 3D Medical Images: Two major brain MRI datasets: BraTS (326 patients, high-grade vs. low-grade tumor classification) and EGD (768 scans, IDH mutation status prediction).

Key Insights and Ideas

  • The Synergy of Losses: The authors found that reconstruction and contrastive learning are complementary. While reconstruction helps the model understand local details, the contrastive loss helps it learn global consistency.
  • Structural Awareness is Key: Adding edge and perceptual losses significantly improved the visual quality of reconstructions and the downstream classification performance.
  • Stability through Decay: A critical "aha moment" involved the edge loss ($L_{edge}$). Because edge maps are sensitive to noise, the authors found that linearly decaying the weight of this loss during training was essential to prevent the loss from exploding and to ensure stable convergence.
Figure 2: Effects of two key parameters. Left: Masking ratio vs. AUC on the BraTS dataset. The AUC and specificity increase with higher masking ratio, peaking at 0.75, after which they begin to decrease. Right: Training loss with Eq. 4 over epochs with and without weight decay of λ2 for Ledge. Without decay (blue), the loss may explode and the training could not converge; With decay (red), training is stable and converges.

Key Results

ViT-AE++ demonstrated consistent improvements across the board. In 2D benchmarks, it outperformed vanilla ViT-AE on every dataset, including a jump from 75.61% to 78.82% accuracy on CIFAR-100.

The most striking results appeared in 3D medical tasks. On the EGD dataset, ViT-AE++ achieved an AUC of 0.846, vastly outperforming established contrastive methods like MoCo-v3 (0.734) and SimSiam (0.741). The ablation study in Table 3 highlights how each added component incrementally boosts the model's performance.

Table 3: Ablation study of the auxiliary loss and contrastive loss on the BraTS dataset.

Limitations and Future Work

The authors note that while ViT backbones are powerful, they are prone to overfitting on smaller medical datasets (like BraTS) compared to traditional ResNets. This suggests that while the training objectives are superior, the architecture itself requires careful tuning or larger data pools to reach its full potential.

Future work will focus on improving the generalizability of these representations, specifically looking at how to make models immune to "domain shifts" caused by different MRI scanners or image acquisition protocols.

Summary updated

Abstract

Self-supervised learning has attracted increasing attention as it learns data-driven representation from data without annotations. Vision transformer-based autoencoder (ViT-AE) by He et al. (2021) is a recent self-supervised learning technique that employs a patch-masking strategy to learn a meaningful latent space. In this paper, we focus on improving ViT-AE (nicknamed ViT-AE++) for a more effective representation of 2D and 3D medical images. We propose two new loss functions to enhance the representation during training. The first loss term aims to improve self-reconstruction by considering the structured dependencies and indirectly improving the representation. The second loss term leverages contrastive loss to optimize the representation from two randomly masked views directly. We extended ViT-AE++ to a 3D fashion for volumetric medical images as an independent contribution. We extensively evaluate ViT-AE++ on both natural images and medical images, demonstrating consistent improvement over vanilla ViT-AE and its superiority over other contrastive learning approaches. Codes are here: https://github.com/chinmay5/vit_ae_plus_plus.git.
BibTeX
@inproceedings{prabhakar2023vit,
  author = {Chinmay Prabhakar and Hongwei Bran Li and Jiancheng Yang and Suprosana Shit and Benedikt Wiestler and Bjoern Menze},
  title = {{ViT-AE++: Improving Vision Transformer Autoencoder for Self-supervised Medical Image Representations}},
  booktitle = {MIDL},
  year = {2023},
  url = {https://openreview.net/forum?id=t4tmvvIIxz},
}
Figure