Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off

Futa Kai Waseda, Ching-Chun Chang, Isao Echizen

Adversarial training (AT) often forces a painful choice between a model's robustness to attacks and its accuracy on clean data. While training on "noisy" adversarial images is the gold standard for defense, it usually tanks the model's performance on normal, clean images.

To mitigate this, researchers have previously tried invariance regularization, which encourages the model to produce the same internal representations for both clean and adversarial versions of an image. However, even this approach suffers from a significant accuracy drop. In this paper, the authors investigate why this happens and introduce AR-AT (Asymmetric Representation-regularized Adversarial Training) to fix the hidden "saboteurs" in standard training.

The Two Culprits of Accuracy Loss

The authors discovered two main reasons why traditional invariance regularization fails to maintain clean accuracy:

  1. Gradient Conflict: The mathematical "pull" to make representations invariant often fights against the "pull" to classify images correctly. This conflicting objective leads to suboptimal convergence, leaving the model confused rather than robust.
  2. The Mixture Distribution Problem: Batch Normalization (BN) layers calculate statistics (mean and variance) based on the data they see. When clean and adversarial images are fed into the same BN layer, the resulting statistics are a "messy middle" that doesn't fit either distribution well.

The AR-AT Framework

To address these issues, the authors propose AR-AT, a novel framework built on three key pillars designed to stabilize training and separate conflicting data distributions.

Architecture of the proposed AR-AT framework, highlighting the predictor, stop-gradient operation, and Split-BN.
  • Asymmetric Regularization (Stop-Gradient): Instead of forcing clean and adversarial representations toward each other symmetrically, AR-AT uses a stop-grad operation on the clean features. This treats the clean representation as a fixed target, preventing the adversarial noise from "corrupting" the clean feature space during optimization. The authors call this the "Purification" Hypothesis: forcing the model to only move adversarial representations toward clean ones preserves discriminative ability.
  • Predictor MLP: Inspired by self-supervised learning architectures like SimSiam, they add a small "predictor" head. This head learns to map the "corrupted" adversarial features back to the "pure" clean features, which further avoids gradient collapse and stabilizes the training process.
  • Split-BatchNorm (Split-BN): To solve the mixture distribution problem, AR-AT uses two separate sets of BN layers during training—one exclusively for clean data and one for adversarial data. This ensures the model learns the distinct statistics of each. During actual use (inference), it reverts to a single set of BNs for practicality.
  • Auto-Balancing: The framework implements a dynamic rule that automatically adjusts the weight of the clean versus adversarial loss based on the model's current training accuracy, significantly reducing the need for manual hyperparameter tuning.

Key Results and Insights

The researchers evaluated AR-AT on standard computer vision benchmarks (CIFAR-10, CIFAR-100, and Imagenette) using ResNet-18 and WideResNet-34-10 architectures. AR-AT consistently outperformed state-of-the-art defenses like TRADES, MART, and LBGAT.

  • Superior Trade-off: On CIFAR-10 (WideResNet), AR-AT achieved 90.89% clean accuracy and 50.77% robustness (AutoAttack). By comparison, the popular TRADES method only managed 84.33% clean accuracy for a similar level of robustness.
  • Computational Efficiency: Despite the extra predictor head, AR-AT remains computationally efficient. It is roughly 1.15x slower than standard AT, making it significantly faster than TRADES (1.51x) or LBGAT (1.50x).
  • Visual Clarity: UMAP visualizations demonstrated that AR-AT produces much tighter, more distinct clusters for different classes compared to naive invariance methods, which often result in overlapping, non-discriminative features.

A New Look at Distillation

Beyond the core framework, the authors provide a fresh perspective on Knowledge Distillation (KD) defenses. They argue that KD-based methods (like LBGAT) succeed largely because they implicitly solve the gradient conflict by using a fixed teacher network as a "stop-gradient" reference, validating the core asymmetric design of AR-AT.

Limitations and Future Work

While highly effective, AR-AT currently relies heavily on the behavior of BatchNorm. It remains unclear how this translates to Transformer-based models (like Vision Transformers) that utilize Layer Normalization instead. Additionally, while the authors recommend regularizing the later stages of the network, the "optimal" layers to apply these constraints can vary depending on the specific model architecture. Future work will likely explore how these asymmetric constraints scale to massive datasets like the full ImageNet-1k.

Summary updated

Abstract

Adversarial training often suffers from a robustness-accuracy trade-off, where achieving high robustness comes at the cost of accuracy. One approach to mitigate this trade-off is leveraging invariance regularization, which encourages model invariance under adversarial perturbations; however, it still leads to accuracy loss. In this work, we closely analyze the challenges of using invariance regularization in adversarial training and understand how to address them. Our analysis identifies two key issues: (1) a "gradient conflict" between invariance and classification objectives, leading to suboptimal convergence, and (2) the mixture distribution problem arising from diverged distributions between clean and adversarial inputs. To address these issues, we propose Asymmetric Representation-regularized Adversarial Training (ARAT), which incorporates asymmetric invariance loss with stop-gradient operation and a predictor to avoid gradient conflict, and a split-BatchNorm (BN) structure to resolve the mixture distribution problem. Our detailed analysis demonstrates that each component effectively addresses the identified issues, offering novel insights into adversarial defense. ARAT shows superiority over existing methods across various settings. Finally, we discuss the implications of our findings to knowledge distillation-based defenses, providing a new perspective on their relative successes.
BibTeX
@inproceedings{waseda2025rethinking,
  author = {Futa Kai Waseda and Ching-Chun Chang and Isao Echizen},
  title = {{Rethinking Invariance Regularization in Adversarial Training to Improve Robustness-Accuracy Trade-off}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=M9SKazbVkJ},
}
Figure