Abstract
Vision Transformers (ViTs) have revolutionized computer vision by leveraging long-range dependency modeling, often surpassing traditional Convolutional Neural Networks (CNNs). However, many architectural variants designed for efficiency or standard accuracy inadvertently compromise model robustness. This paper systematically re-evaluates ViT design principles through the lens of robustness. By identifying and combining robust design components, the authors propose the Robust Vision Transformer (RVT). Enhanced by two novel techniques—Position-Aware Attention Rescaling (PAAS) and Patch-wise Augmentation—RVT achieves state-of-the-art performance on multiple robustness benchmarks, including ImageNet-C and ImageNet-Sketch, without sacrificing computational efficiency.
The Robustness Gap in Vision Transformers
While the research community has historically prioritized the trade-off between standard accuracy and computational cost (FLOPs), models often remain "brittle" when exposed to real-world noise, adversarial attacks, or distribution shifts. Popular variants like Swin or PVT introduce architectural changes that improve efficiency but can destroy the inherent robustness of the Transformer. This study aims to demystify these trade-offs, providing a blueprint for building resilient models ready for deployment in unpredictable environments.
Methodology: Building the RVT
The authors deconstructed the ViT architecture into four core components—patch embedding, position embedding, Transformer blocks, and the classification head—to identify robustness bottlenecks.
Key Architectural Refinements
- Convolutional Stems: Replacing simple linear patch embeddings with a convolutional stem significantly improves the capture of low-level features like edges, which is vital for robustness.
- Global Average Pooling (GAP): Replacing the traditional
[CLS]token with GAP enhances translation invariance and resilience to spatial shifts. - Optimized Attention Heads: The study found a "sweet spot" for the number of attention heads (e.g., 8 heads for smaller models), providing diverse "views" of input data that bolster stability.
Novel Training Techniques
- Position-Aware Attention Rescaling (PAAS): Instead of simple additive position embeddings, PAAS uses a learnable matrix to rescale attention scores. This acts as a soft mask, filtering out noisy correlations and making the model less susceptible to adversarial distractions.
- Patch-wise Augmentation: Unlike standard image-level augmentation, this technique applies different random transformations to individual patches. This forces the model to learn more diverse inter-patch relationships and reduces overfitting to global patterns.
Experimental Results and Insights
RVT was evaluated against a comprehensive suite of benchmarks, including ImageNet-C (common corruptions), ImageNet-R (renditions), and white-box adversarial attacks (FGSM/PGD).
As shown in the performance plots, RVT consistently maintains a superior Pareto front for both standard and robust accuracy compared to models like DeiT, Swin, and PiT.
- ImageNet-C: RVT-S achieved 49.4 mCE* (mean Corruption Error), securing the top spot on the leaderboard.
- Adversarial Strength: Under FGSM attacks, RVT-S reached 51.3% accuracy, a significant jump from the 40.7% achieved by the baseline DeiT-S.
- Generalization: On ImageNet-Sketch and ImageNet-R, RVT-S set new state-of-the-art benchmarks with 34.7% and 47.7%* accuracy, respectively.
Conclusion
The Robust Vision Transformer (RVT) demonstrates that robustness does not have to come at the cost of performance. By integrating convolutional stems, multi-stage blocks, and novel rescaling techniques, RVT overcomes the fragility of standard ViTs. While techniques like patch-wise augmentation show marginal returns at very high model capacities, the overall design principles established here provide a robust foundation for the next generation of vision models.
Summary updated