Introduction: The Fragility of Global Attention
Vision Transformers (ViTs) have fundamentally shifted the landscape of computer vision, leveraging self-attention to capture long-range dependencies across images. However, this global connectivity introduces a unique vulnerability: the "Achilles' heel" of patch-based attention. Because every patch interacts with every other patch, minor corruptions or occlusions in a small subset of input patches (e.g., 10% of the image) can propagate through the network, causing internal attention mechanisms to collapse and model confidence to plummet.
Traditional defenses, such as Adversarial Training (AT), involve generating worst-case pixel perturbations during training. While effective, AT is computationally prohibitive for large-scale models. This paper introduces RSPC (Reducing Sensitivity to Patch Corruptions), a training framework designed to identify and mitigate these specific structural vulnerabilities in ViTs with significantly lower overhead.
Methodology: The RSPC Mini-Max Framework
The RSPC approach treats robustness as a mini-max game between two competing components: a lightweight corruption model and the primary Vision Transformer.
1. The Patch Corruption Model
Rather than perturbing individual pixels, the RSPC "attacker" is a lightweight network that learns to select the most informative patches for occlusion. By using a Straight Through Estimator (STE), the model makes the discrete choice of patch selection differentiable. The goal of this attacker is to maximize the disruption of the main model's internal feature representations.
2. Feature Alignment Loss
As the attacker identifies vulnerable patches, the primary ViT is trained to remain invariant to these disruptions. This is achieved through a feature alignment loss, which penalizes the network if the intermediate representations of a corrupted image deviate from those of the original, clean image. By forcing the model to ignore the loss of specific patches, the network develops a more stable and reliable attention mechanism.
Experimental Evaluation
The researchers evaluated RSPC across a suite of rigorous robustness benchmarks, including ImageNet-C (common corruptions), ImageNet-A (natural adversarial examples), and ImageNet-P (perturbation stability).
Key Results
- Superior Robustness: RSPC consistently outperformed state-of-the-art baselines across various architectures, including DeiT, Swin, RVT, and FAN. On ImageNet-C, the RSPC-FAN-B-Hybrid model achieved a 1.6% reduction in mean Corruption Error (mCE) over the baseline.
- Computational Efficiency: RSPC is remarkably faster than standard adversarial training. The corruption model is approximately 80x faster than a single iteration of Projected Gradient Descent (PGD), making robust training feasible for large-scale datasets like ImageNet.
- Minimal Accuracy Trade-off: Unlike many robustness techniques that sacrifice performance on clean data, RSPC often maintains or slightly improves clean accuracy while providing substantial gains in robustness.
Discussion: Efficiency and Attention Stability
A central insight of this work is that simple patch occlusion serves as a highly effective proxy for expensive adversarial perturbations. The corruption model naturally learns to target the most informative regions of an image—such as the eyes of an animal—because these are the patches the self-attention mechanism relies on most heavily for classification.
Furthermore, RSPC-trained models exhibit significantly higher "Cosine Similarity" between their attention maps for clean and noisy images. This suggests that the training process successfully stabilizes the internal logic of the transformer, preventing the "attention shift" that typically occurs when noise is introduced.
Conclusion
RSPC offers a scalable and efficient path toward robust Vision Transformers. By shifting the focus from pixel-level noise to patch-level structural vulnerabilities, the method achieves state-of-the-art robustness with a fraction of the computational cost of traditional methods. Future work may explore extending this "vulnerability-seeking" approach to complex geometric transformations and further refining the balance between occlusion ratios and alignment loss.
Summary updated