Improving Robustness of Vision Transformers by Reducing Sensitivity to Patch Corruptions

Yong Guo, David Stutz, Bernt Schiele

Introduction: The Fragility of Global Attention

Vision Transformers (ViTs) have fundamentally shifted the landscape of computer vision, leveraging self-attention to capture long-range dependencies across images. However, this global connectivity introduces a unique vulnerability: the "Achilles' heel" of patch-based attention. Because every patch interacts with every other patch, minor corruptions or occlusions in a small subset of input patches (e.g., 10% of the image) can propagate through the network, causing internal attention mechanisms to collapse and model confidence to plummet.

Traditional defenses, such as Adversarial Training (AT), involve generating worst-case pixel perturbations during training. While effective, AT is computationally prohibitive for large-scale models. This paper introduces RSPC (Reducing Sensitivity to Patch Corruptions), a training framework designed to identify and mitigate these specific structural vulnerabilities in ViTs with significantly lower overhead.

Figure 3: Comparison of different patch-level corruptions and their impact on model confidence. Note how adversarial perturbations and occlusions significantly degrade performance compared to the clean image.

Methodology: The RSPC Mini-Max Framework

The RSPC approach treats robustness as a mini-max game between two competing components: a lightweight corruption model and the primary Vision Transformer.

1. The Patch Corruption Model

Rather than perturbing individual pixels, the RSPC "attacker" is a lightweight network that learns to select the most informative patches for occlusion. By using a Straight Through Estimator (STE), the model makes the discrete choice of patch selection differentiable. The goal of this attacker is to maximize the disruption of the main model's internal feature representations.

2. Feature Alignment Loss

As the attacker identifies vulnerable patches, the primary ViT is trained to remain invariant to these disruptions. This is achieved through a feature alignment loss, which penalizes the network if the intermediate representations of a corrupted image deviate from those of the original, clean image. By forcing the model to ignore the loss of specific patches, the network develops a more stable and reliable attention mechanism.

Experimental Evaluation

The researchers evaluated RSPC across a suite of rigorous robustness benchmarks, including ImageNet-C (common corruptions), ImageNet-A (natural adversarial examples), and ImageNet-P (perturbation stability).

Key Results

  • Superior Robustness: RSPC consistently outperformed state-of-the-art baselines across various architectures, including DeiT, Swin, RVT, and FAN. On ImageNet-C, the RSPC-FAN-B-Hybrid model achieved a 1.6% reduction in mean Corruption Error (mCE) over the baseline.
  • Computational Efficiency: RSPC is remarkably faster than standard adversarial training. The corruption model is approximately 80x faster than a single iteration of Projected Gradient Descent (PGD), making robust training feasible for large-scale datasets like ImageNet.
  • Minimal Accuracy Trade-off: Unlike many robustness techniques that sacrifice performance on clean data, RSPC often maintains or slightly improves clean accuracy while providing substantial gains in robustness.

Discussion: Efficiency and Attention Stability

A central insight of this work is that simple patch occlusion serves as a highly effective proxy for expensive adversarial perturbations. The corruption model naturally learns to target the most informative regions of an image—such as the eyes of an animal—because these are the patches the self-attention mechanism relies on most heavily for classification.

Furthermore, RSPC-trained models exhibit significantly higher "Cosine Similarity" between their attention maps for clean and noisy images. This suggests that the training process successfully stabilizes the internal logic of the transformer, preventing the "attention shift" that typically occurs when noise is introduced.

Conclusion

RSPC offers a scalable and efficient path toward robust Vision Transformers. By shifting the focus from pixel-level noise to patch-level structural vulnerabilities, the method achieves state-of-the-art robustness with a fraction of the computational cost of traditional methods. Future work may explore extending this "vulnerability-seeking" approach to complex geometric transformations and further refining the balance between occlusion ratios and alignment loss.

Summary updated

Abstract

Despite their success, vision transformers still remain vulnerable to image corruptions, such as noise or blur. Indeed, we find that the vulnerability mainly stems from the unstable self-attention mechanism, which is inherently built upon patch-based inputs and often becomes overly sensitive to the corruptions across patches. For example, when we only occlude a small number of patches with random noise (e.g., 10%), these patch corruptions would lead to severe accuracy drops and greatly distract intermediate attention layers. To address this, we propose a new training method that improves the robustness of transformers from a new perspective - reducing sensitivity to patch corruptions (RSPC). Specifically, we first identify and occlude/corrupt the most vulnerable patches and then explicitly reduce sensitivity to them by aligning the intermediate features between clean and corrupted examples. We highlight that the construction of patch corruptions is learned adversarially to the following feature alignment process, which is particularly effective and essentially different from existing methods. In experiments, our RSPC greatly improves the stability of attention layers and consistently yields better robustness on various benchmarks, including CIFAR-10/100-C, ImageNet-A, ImageNet-C, and ImageNet-P.
BibTeX
@inproceedings{guo2023improving,
  author = {Yong Guo and David Stutz and Bernt Schiele},
  title = {{Improving Robustness of Vision Transformers by Reducing Sensitivity to Patch Corruptions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00400},
}
Figure