Introduction
Vision Transformers (ViTs) have emerged as powerful alternatives to Convolutional Neural Networks (CNNs), often achieving superior performance on standard benchmarks. However, ViTs exhibit a significant vulnerability: they are highly sensitive to image corruptions and adversarial perturbations. Existing methods to improve robustness typically necessitate computationally expensive retraining or fundamental architectural modifications. This paper introduces SATA (Spatial Autocorrelation Token Analysis), a "plug-and-play" module that enhances ViT robustness and accuracy without requiring any retraining.
The Stability of Spatial Autocorrelation
A key observation driving this work is the relative instability of self-attention maps compared to spatial autocorrelation under noise. While the similarity between attention maps of clean and corrupted images can drop significantly in deeper layers, the spatial autocorrelation scores—measuring how much a token's features are influenced by its neighbors—remain remarkably stable. This stability suggests that spatial autocorrelation can serve as a more reliable signal for feature processing in noisy environments than the attention mechanism itself.
The SATA Framework
SATA is integrated into the later blocks of a ViT, positioned between the self-attention layer and the Feed-Forward Network (FFN). The module operates through a multi-stage process:
- Spatial Autocorrelation Analysis: Utilizing Moran’s I, a geographical statistic, SATA calculates a score for each token. The model's existing attention map is used as the spatial weight matrix, ensuring computational efficiency.
- Token Splitting: Tokens are partitioned based on their autocorrelation scores. Tokens with scores near the mean are considered stable, while those with extreme scores are identified as potentially redundant or non-informative.
- Token Grouping and Merging: Instead of simple pruning, SATA employs a bipartite matching algorithm to merge similar redundant tokens. This reduces the token count passed to the FFN, thereby decreasing computational overhead.
- Residual Integration: To maintain the expected data dimensions, original tokens are concatenated back following the FFN stage.
Experimental Results
SATA was evaluated on ImageNet-1K and various robustness benchmarks, including ImageNet-C (common corruptions), ImageNet-A (natural adversarial examples), and ImageNet-R (renditions).
- Accuracy and Efficiency: A ViT-Base model with SATA achieved a Top-1 accuracy of 94.9% on ImageNet-1K while reducing total GFLOPs. This demonstrates that the module can improve performance while simultaneously lowering computational costs.
- Corruption Resilience: On ImageNet-C, SATA-B significantly reduced the Mean Corruption Error (mCE) to 13.6%, a substantial improvement over vanilla baselines which often hover around 40-50%.
- Adversarial and OOD Performance: SATA demonstrated a 20% improvement in accuracy against FGSM adversarial attacks and achieved 79.2% accuracy on ImageNet-R, outperforming many specialized robust models that require heavy training.
Conclusion
SATA provides a highly effective, inference-only solution for enhancing the robustness of Vision Transformers. By leveraging the stability of spatial autocorrelation, it intelligently compresses redundant information, leading to simultaneous gains in accuracy, robustness, and efficiency. Future work may extend this approach to window-based transformers and other modalities like natural language processing.
Summary updated