Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off

Futa Waseda, Isao Echizen

TL;DR: Researchers have developed AR-AT (Asymmetric Representation-regularized Adversarial Training), a new approach that significantly improves the balance between a model's "clean" accuracy and its resistance to adversarial attacks. By identifying and fixing "gradient conflicts" and batch normalization issues, AR-AT outperforms existing state-of-the-art defenses while being more computationally efficient than popular alternatives like TRADES.

Abstract

Although adversarial training has been the state-of-the-art approach to defend against adversarial examples (AEs), it suffers from a robustness-accuracy trade-off, where high robustness is achieved at the cost of clean accuracy. In this work, we leverage invariance regularization on latent representations to learn discriminative yet adversarially invariant representations, aiming to mitigate this trade-off. We analyze two key issues in representation learning with invariance regularization: (1) a "gradient conflict" between invariance loss and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs. To address these issues, we propose Asymmetrically Representation-regularized Adversarial Training (AR-AT), which incorporates asymmetric invariance loss with stop-gradient operation and a predictor to improve the convergence, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem. Our method significantly improves the robustness-accuracy trade-off by learning adversarially invariant representations without sacrificing discriminative ability. Furthermore, we discuss the relevance of our findings to knowledge-distillation-based defense methods, contributing to a deeper understanding of their relative successes.

1. Introduction

Deep neural networks are notoriously fragile. A tiny, invisible-to-humans perturbation (an adversarial example) can cause a state-of-the-art model to misclassify an image completely. To defend against this, researchers use Adversarial Training (AT), where the model is trained on these "attack" images. However, AT comes with a painful "robustness-accuracy trade-off": the more robust you make the model against attacks, the worse it performs on normal, clean images. A common fix is invariance regularization—forcing the model to produce the same internal representations for both clean and attacked versions of an image. While this helps, it still degrades clean accuracy.

2. The AR-AT Framework

The authors introduce AR-AT, which stands for Asymmetric Representation-regularized Adversarial Training. It addresses the trade-off through three key architectural innovations:

Comparison of AR-AT (f) with existing invariance-based methods like MART (b), TRADES (c), and ARREST (e).
  1. Asymmetric Invariance Loss (with Stop-Gradient): Standard methods try to pull clean and adversarial representations toward each other. The authors found this "corrupts" the clean representations. AR-AT uses a stop-gradient operation on the clean features. This forces the adversarial features to move toward the clean ones ("purification") without allowing the clean features to be dragged toward the noisy adversarial ones.
  2. Predictor MLP: A small "predictor" head (a multi-layer perceptron) maps the adversarial representation back to the clean one. This acts as a stabilizer, preventing the model from overreacting to the high variance of adversarial perturbations during training.
  3. Split-BatchNorm (BN): Clean and adversarial images have different statistical distributions. Using the same Batch Normalization layers for both creates a "mixture distribution problem" that confuses the model. AR-AT uses separate BN layers for clean and adversarial data during training, then merges them for inference.

3. Key Insights and Analysis

Gradient Conflict: Corruption vs. Purification

The authors mathematically decomposed the standard invariance loss and discovered that half of the loss function was actually trying to make clean images look more like adversarial ones. By applying a stop-gradient, they isolated the "purification" effect, which improved clean accuracy.

Solving the BN Identity Crisis

Sharing BN statistics between clean and noisy data leads to high variance in the model's running means. Split-BN stabilizes these statistics, which is crucial for the model to learn effectively.

A New Perspective on Knowledge Distillation (KD)

Many top-tier defenses use a "teacher" model to guide a student (KD). The authors realized these KD methods work because the teacher acts as a natural "stop-gradient" by providing a fixed reference point that doesn't get corrupted by the student's noisy training process. AR-AT achieves similar benefits within a single model.

AR-AT (f) compared to teacher-based methods like LGBAT (d) and ARREST (e). The pre-trained teacher in (e) provides a fixed reference similar to the stop-gradient in AR-AT.

4. Experimental Results

AR-AT consistently pushed the Pareto frontier of the robustness-accuracy trade-off across standard benchmarks:

  • CIFAR-10 (ResNet-18): AR-AT achieved 87.82% clean accuracy and 49.02% robust accuracy (against AutoAttack). For context, the popular TRADES method only hit 81.25% clean accuracy for a similar level of robustness.
  • Efficiency: AR-AT is faster than many competitors. While standard AT is the fastest (1.0x), AR-AT takes about 1.15x the time, whereas TRADES and LBGAT take 1.5x longer.
  • SWA Boost: Combining AR-AT with Stochastic Weight Averaging (SWA) further boosted performance, reaching nearly 54% robust accuracy on CIFAR-10 with a WideResNet-34-10 architecture.

5. Conclusion

AR-AT significantly improves the robustness-accuracy trade-off by learning adversarially invariant representations without sacrificing discriminative ability. By resolving gradient conflicts and the mixture distribution problem, it offers a more efficient and effective path toward robust deep learning. Future work will explore extending these concepts to architectures utilizing Layer Normalization, such as Transformers.

Summary updated

Abstract

Although adversarial training has been the state-of-the-art approach to defend against adversarial examples (AEs), it suffers from a robustness-accuracy trade-off, where high robustness is achieved at the cost of clean accuracy. In this work, we leverage invariance regularization on latent representations to learn discriminative yet adversarially invariant representations, aiming to mitigate this trade-off. We analyze two key issues in representation learning with invariance regularization: (1) a "gradient conflict" between invariance loss and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions of clean and adversarial inputs. To address these issues, we propose Asymmetrically Representation-regularized Adversarial Training (AR-AT), which incorporates asymmetric invariance loss with stop-gradient operation and a predictor to improve the convergence, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem. Our method significantly improves the robustness-accuracy trade-off by learning adversarially invariant representations without sacrificing discriminative ability. Furthermore, we discuss the relevance of our findings to knowledge-distillation-based defense methods, contributing to a deeper understanding of their relative successes.
BibTeX
@inproceedings{waseda2025rethinking,
  author = {Futa Waseda and Isao Echizen},
  title = {{Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=M9SKazbVkJ},
}
Figure