Trade-off between Robustness and Accuracy of Vision Transformers

Yanxi Li, Chang XuCVPR 2023 · DOI

Abstract

Vision Transformers (ViTs) face a significant trade-off: increasing robustness to adversarial attacks typically degrades accuracy on clean images. This paper introduces TORA-ViTs (Trade-off between Robustness and Accuracy of Vision Transformers), a framework that decouples these objectives using a dual-adapter system and a gated fusion module. By freezing the pre-trained ViT backbone and training lightweight adapters, TORA-ViTs provide an efficient way to navigate the robustness-accuracy frontier via a single tunable hyperparameter, $\lambda$.

Introduction

The "no free lunch" theorem in computer vision suggests that models often rely on "non-robust predictive features"—patterns that aid classification but are easily disrupted by adversarial perturbations. While ViTs are inherently more robust than traditional Convolutional Neural Networks (CNNs), they still suffer from this fundamental friction. Existing methods typically attempt to learn both robust and predictive features within a single set of weights, often leading to suboptimal performance in both domains.

Methodology: The TORA-ViTs Framework

TORA-ViTs address the robustness-accuracy trade-off by treating them as distinct goals handled by specialized, modular components.

Dual Adapters

Instead of fine-tuning the entire network, TORA-ViTs insert two small, trainable adapters after the MLP layer of each transformer block:

  1. Accuracy Adapter ($\psi_{A,l}$): Optimized for high performance on clean, natural images.
  2. Robustness Adapter ($\psi_{R,l}$): Optimized for resilience against adversarial attacks.

The main ViT backbone remains frozen, which significantly reduces the computational cost compared to full adversarial training.

The TORA-ViT architecture featuring dual adapters and a gated fusion module.

Gated Fusion Module

The core innovation is the Gated Fusion Module ($\phi_l$), which dynamically balances the contributions from the two adapters. It employs an attention mechanism where the output of the frozen ViT block serves as a "query," while the adapter outputs serve as "keys" and "values."

Matrix operations and softmax comparison for the gated fusion mechanism.

Training Strategy

The system is trained in two phases. First, the adapters are trained independently. Second, the fusion module is fine-tuned using a joint objective function weighted by a trade-off ratio, $\lambda$. This allows users to shift the model's focus between accuracy (low $\lambda$) and robustness (high $\lambda$).

Experimental Results

TORA-ViTs were evaluated on ImageNet-1K and several robustness benchmarks, including adversarial attacks (FGSM, PGD) and natural corruptions (ImageNet-A, -R, -C).

Performance Benchmarks

In a balanced configuration ($\lambda = 0.5$), TORA-ViTs achieved:

  • Clean Accuracy: 83.7% on ImageNet.
  • Adversarial Robustness: 54.7% (FGSM) and 38.0% (PGD).
  • Distribution Shifts: 39.2% on ImageNet-A and 56.3% on ImageNet-R.
  • Corruption Resistance: A mean Corruption Error (mCE) of 34.4% on ImageNet-C.

When tuned for maximum robustness ($\lambda = 0.9$), the model's FGSM accuracy reached 74.2%, demonstrating the framework's flexibility.

Qualitative Insights

Visualization of attention maps revealed that the Accuracy Adapter focuses on contextual information, whereas the Robustness Adapter concentrates on the primary object. This supports the hypothesis that context, while predictive, is often non-robust.

Conclusion

TORA-ViTs offer a scalable and efficient solution to the robustness-accuracy trade-off in Vision Transformers. By modularizing the learning of predictive and robust features, the framework allows for post-training adjustment of model behavior, outperforming previous state-of-the-art methods with minimal parameter overhead.

Summary updated

BibTeX
@inproceedings{li2023trade,
  author = {Yanxi Li and Chang Xu},
  title = {{Trade-off between Robustness and Accuracy of Vision Transformers}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00730},
}
Figure