Rethinking the Design Principles of Robust Vision Transformer

Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Shaokai Ye, Yuan He, Hui Xue

Abstract

Vision Transformers (ViTs) have revolutionized computer vision by leveraging long-range dependency modeling, often surpassing traditional Convolutional Neural Networks (CNNs). However, many architectural variants designed for efficiency or standard accuracy inadvertently compromise model robustness. This paper systematically re-evaluates ViT design principles through the lens of robustness. By identifying and combining robust design components, the authors propose the Robust Vision Transformer (RVT). Enhanced by two novel techniques—Position-Aware Attention Rescaling (PAAS) and Patch-wise Augmentation—RVT achieves state-of-the-art performance on multiple robustness benchmarks, including ImageNet-C and ImageNet-Sketch, without sacrificing computational efficiency.

The Robustness Gap in Vision Transformers

While the research community has historically prioritized the trade-off between standard accuracy and computational cost (FLOPs), models often remain "brittle" when exposed to real-world noise, adversarial attacks, or distribution shifts. Popular variants like Swin or PVT introduce architectural changes that improve efficiency but can destroy the inherent robustness of the Transformer. This study aims to demystify these trade-offs, providing a blueprint for building resilient models ready for deployment in unpredictable environments.

Methodology: Building the RVT

The authors deconstructed the ViT architecture into four core components—patch embedding, position embedding, Transformer blocks, and the classification head—to identify robustness bottlenecks.

Comparison of ViT and RVT architectures alongside factorized blocks.

Key Architectural Refinements

  1. Convolutional Stems: Replacing simple linear patch embeddings with a convolutional stem significantly improves the capture of low-level features like edges, which is vital for robustness.
  2. Global Average Pooling (GAP): Replacing the traditional [CLS] token with GAP enhances translation invariance and resilience to spatial shifts.
  3. Optimized Attention Heads: The study found a "sweet spot" for the number of attention heads (e.g., 8 heads for smaller models), providing diverse "views" of input data that bolster stability.

Novel Training Techniques

  • Position-Aware Attention Rescaling (PAAS): Instead of simple additive position embeddings, PAAS uses a learnable matrix to rescale attention scores. This acts as a soft mask, filtering out noisy correlations and making the model less susceptible to adversarial distractions.
  • Patch-wise Augmentation: Unlike standard image-level augmentation, this technique applies different random transformations to individual patches. This forces the model to learn more diverse inter-patch relationships and reduces overfitting to global patterns.

Experimental Results and Insights

RVT was evaluated against a comprehensive suite of benchmarks, including ImageNet-C (common corruptions), ImageNet-R (renditions), and white-box adversarial attacks (FGSM/PGD).

Standard and Robust Accuracy vs. FLOPS across various Vision Transformer models.

As shown in the performance plots, RVT consistently maintains a superior Pareto front for both standard and robust accuracy compared to models like DeiT, Swin, and PiT.

  • ImageNet-C: RVT-S achieved 49.4 mCE* (mean Corruption Error), securing the top spot on the leaderboard.
  • Adversarial Strength: Under FGSM attacks, RVT-S reached 51.3% accuracy, a significant jump from the 40.7% achieved by the baseline DeiT-S.
  • Generalization: On ImageNet-Sketch and ImageNet-R, RVT-S set new state-of-the-art benchmarks with 34.7% and 47.7%* accuracy, respectively.

Conclusion

The Robust Vision Transformer (RVT) demonstrates that robustness does not have to come at the cost of performance. By integrating convolutional stems, multi-stage blocks, and novel rescaling techniques, RVT overcomes the fragility of standard ViTs. While techniques like patch-wise augmentation show marginal returns at very high model capacities, the overall design principles established here provide a robust foundation for the next generation of vision models.

Summary updated

Abstract

Recent advances on Vision Transformers (ViT) have shown that self-attention-based networks, which take advantage of long-range dependencies modeling ability, surpassed traditional convolution neural networks (CNNs) in most vision tasks. To further expand the applicability for computer vision, many improved variants are proposed to re-design the Transformer architecture by considering the superiority of CNNs, i.e., locality, translation invariance, for better performance. However, these methods only consider the standard accuracy or computation cost of the model. In this paper, we rethink the design principles of ViTs based on the robustness. We found some design components greatly harm the robustness and generalization ability of ViTs while some others are beneficial. By combining the robust design components, we propose Robust Vision Transformer (RVT). RVT is a new vision transformer, which has superior performance and strong robustness. We further propose two new plug-and-play techniques called position-aware attention rescaling and patch-wise augmentation to train our RVT. The experimental results on ImageNet and six robustness benchmarks show the advanced robustness and generalization ability of RVT compared with previous Transformers and state-of-the-art CNNs. Our RVT-S* also achieves Top-1 rank on multiple robustness leaderboards including ImageNet-C and ImageNet-Sketch. The code will be available at https://github.com/vtddggg/Robust-Vision-Transformer.
BibTeX
@inproceedings{mao2021rethinking,
  author = {Xiaofeng Mao and Gege Qi and Yuefeng Chen and Xiaodan Li and Shaokai Ye and Yuan He and Hui Xue},
  title = {{Rethinking the Design Principles of Robust Vision Transformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr52688.2022.01173},
}
Figure