Abstract
Vision Transformers (ViTs) face a significant trade-off: increasing robustness to adversarial attacks typically degrades accuracy on clean images. This paper introduces TORA-ViTs (Trade-off between Robustness and Accuracy of Vision Transformers), a framework that decouples these objectives using a dual-adapter system and a gated fusion module. By freezing the pre-trained ViT backbone and training lightweight adapters, TORA-ViTs provide an efficient way to navigate the robustness-accuracy frontier via a single tunable hyperparameter, $\lambda$.
Introduction
The "no free lunch" theorem in computer vision suggests that models often rely on "non-robust predictive features"—patterns that aid classification but are easily disrupted by adversarial perturbations. While ViTs are inherently more robust than traditional Convolutional Neural Networks (CNNs), they still suffer from this fundamental friction. Existing methods typically attempt to learn both robust and predictive features within a single set of weights, often leading to suboptimal performance in both domains.
Methodology: The TORA-ViTs Framework
TORA-ViTs address the robustness-accuracy trade-off by treating them as distinct goals handled by specialized, modular components.
Dual Adapters
Instead of fine-tuning the entire network, TORA-ViTs insert two small, trainable adapters after the MLP layer of each transformer block:
- Accuracy Adapter ($\psi_{A,l}$): Optimized for high performance on clean, natural images.
- Robustness Adapter ($\psi_{R,l}$): Optimized for resilience against adversarial attacks.
The main ViT backbone remains frozen, which significantly reduces the computational cost compared to full adversarial training.
Gated Fusion Module
The core innovation is the Gated Fusion Module ($\phi_l$), which dynamically balances the contributions from the two adapters. It employs an attention mechanism where the output of the frozen ViT block serves as a "query," while the adapter outputs serve as "keys" and "values."
Training Strategy
The system is trained in two phases. First, the adapters are trained independently. Second, the fusion module is fine-tuned using a joint objective function weighted by a trade-off ratio, $\lambda$. This allows users to shift the model's focus between accuracy (low $\lambda$) and robustness (high $\lambda$).
Experimental Results
TORA-ViTs were evaluated on ImageNet-1K and several robustness benchmarks, including adversarial attacks (FGSM, PGD) and natural corruptions (ImageNet-A, -R, -C).
Performance Benchmarks
In a balanced configuration ($\lambda = 0.5$), TORA-ViTs achieved:
- Clean Accuracy: 83.7% on ImageNet.
- Adversarial Robustness: 54.7% (FGSM) and 38.0% (PGD).
- Distribution Shifts: 39.2% on ImageNet-A and 56.3% on ImageNet-R.
- Corruption Resistance: A mean Corruption Error (mCE) of 34.4% on ImageNet-C.
When tuned for maximum robustness ($\lambda = 0.9$), the model's FGSM accuracy reached 74.2%, demonstrating the framework's flexibility.
Qualitative Insights
Visualization of attention maps revealed that the Accuracy Adapter focuses on contextual information, whereas the Robustness Adapter concentrates on the primary object. This supports the hypothesis that context, while predictive, is often non-robust.
Conclusion
TORA-ViTs offer a scalable and efficient solution to the robustness-accuracy trade-off in Vision Transformers. By modularizing the learning of predictive and robust features, the framework allows for post-training adjustment of model behavior, outperforming previous state-of-the-art methods with minimal parameter overhead.
Summary updated