Self-supervised learning (SSL) has become a cornerstone of modern computer vision, allowing models to learn rich representations from vast amounts of unlabeled data. In the medical domain, where expert annotations are expensive and scarce, SSL is particularly transformative. This paper introduces ViT-AE++, an evolution of the Vision Transformer Autoencoder (ViT-AE) that combines the best of reconstruction-based learning and contrastive learning to master both 2D and 3D medical image representations.
TL;DR
ViT-AE++ enhances the standard Vision Transformer Autoencoder by integrating auxiliary structural losses (perceptual and edge-aware) and a contrastive learning objective. This hybrid approach significantly improves representation quality for medical imaging, outperforming vanilla ViT-AE and standalone contrastive methods on both 2D X-rays and 3D MRI volumes.
What Problem Does This Paper Address?
While Vision Transformer Autoencoders (like MAE) have shown great promise by learning to reconstruct masked image patches, they suffer from two main drawbacks in the medical context:
- Pixel-wise Limitations: Standard models typically use a simple L2 (mean-squared error) loss for reconstruction. This ignores the "structured dependencies"—the complex textures and morphologies—that are vital for medical diagnosis.
- Indirect Learning: In a pure autoencoder, the latent representation is learned indirectly as a byproduct of reconstruction. There is no explicit objective to ensure that two different views of the same anatomy result in similar feature embeddings.
ViT-AE++ tackles these gaps by forcing the model to care about edges and high-level features while explicitly aligning representations through contrastive learning.
Methods
The authors propose a multi-faceted loss function to train a 3D Vision Transformer. The framework, illustrated in Figure 1, uses a "shared encoder" architecture to process two different augmented and masked versions of the same input.
The total loss is a combination of four terms:
- Pixel-wise Reconstruction ($L_{rec}$): The standard L2 loss for patch recovery.
- Perceptual Loss ($L_{per}$): Uses a pre-trained VGG network to compare high-level features of 2D slices, ensuring the model captures semantic structure.
- Edge-aware Loss ($L_{edge}$): A novel 3D Sobel filter-based loss that penalizes discrepancies in gradients across axial, coronal, and sagittal planes.
- Contrastive Loss ($L_{CL}$): Borrowing from the SimSiam paradigm, this term directly maximizes the similarity between the latent representations of two different masked views of the same image.
Datasets
The model was rigorously tested across natural and medical domains:
- 2D Natural Images: CIFAR-10, CIFAR-100, and Tiny ImageNet-100.
- 2D Medical Images: A Chest X-ray dataset for pneumonia identification.
- 3D Medical Images: Two major brain MRI datasets: BraTS (326 patients, high-grade vs. low-grade tumor classification) and EGD (768 scans, IDH mutation status prediction).
Key Insights and Ideas
- The Synergy of Losses: The authors found that reconstruction and contrastive learning are complementary. While reconstruction helps the model understand local details, the contrastive loss helps it learn global consistency.
- Structural Awareness is Key: Adding edge and perceptual losses significantly improved the visual quality of reconstructions and the downstream classification performance.
- Stability through Decay: A critical "aha moment" involved the edge loss ($L_{edge}$). Because edge maps are sensitive to noise, the authors found that linearly decaying the weight of this loss during training was essential to prevent the loss from exploding and to ensure stable convergence.
Key Results
ViT-AE++ demonstrated consistent improvements across the board. In 2D benchmarks, it outperformed vanilla ViT-AE on every dataset, including a jump from 75.61% to 78.82% accuracy on CIFAR-100.
The most striking results appeared in 3D medical tasks. On the EGD dataset, ViT-AE++ achieved an AUC of 0.846, vastly outperforming established contrastive methods like MoCo-v3 (0.734) and SimSiam (0.741). The ablation study in Table 3 highlights how each added component incrementally boosts the model's performance.
Limitations and Future Work
The authors note that while ViT backbones are powerful, they are prone to overfitting on smaller medical datasets (like BraTS) compared to traditional ResNets. This suggests that while the training objectives are superior, the architecture itself requires careful tuning or larger data pools to reach its full potential.
Future work will focus on improving the generalizability of these representations, specifically looking at how to make models immune to "domain shifts" caused by different MRI scanners or image acquisition protocols.
Summary updated