Bag of Tricks for Adversarial Training

Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, Jun Zhu

Abstract

Adversarial training (AT) is a cornerstone of robust deep learning, yet the field is often clouded by inconsistent hyperparameter settings that obscure true algorithmic progress. This paper systematically evaluates the impact of various training "tricks"—such as weight decay, batch normalization modes, and activation functions—on the robustness of neural networks. By standardizing these overlooked implementation details, we demonstrate that classic methods like PGD-AT and TRADES can be revitalized to achieve state-of-the-art performance on benchmarks like CIFAR-10.

Introduction

The rapid development of new adversarial defense algorithms has led to a surprising observation: older, simpler methods often outperform newer variants when properly tuned. This suggests that many reported improvements in the literature may stem from hidden hyperparameter optimizations rather than fundamental algorithmic breakthroughs. This study isolates these confounding variables to establish a rigorous baseline for adversarial robustness.

The Critical Role of Weight Decay

A primary finding of this work is the extreme sensitivity of robust accuracy to weight decay. While standard "clean" accuracy is relatively robust to minor changes in this hyperparameter, adversarial robustness (measured by PGD-10 accuracy) is highly dependent on it. For instance, increasing weight decay from $1 \times 10^{-4}$ to $5 \times 10^{-4}$ can improve robust accuracy by as much as 7% on CIFAR-10. Furthermore, higher weight decay values appear to mitigate the "robust overfitting" phenomenon, where robust accuracy drops significantly after the first learning rate decay.

Impact of weight decay on accuracy and training dynamics. (a) Sensitivity of clean and PGD-10 accuracy to weight decay values. (b) Training curves showing the effect of weight decay on robust accuracy over epochs.

Optimizing Batch Normalization and Architecture

Beyond weight decay, the mode of Batch Normalization (BN) during the generation of adversarial training examples plays a vital role. Using the eval mode for BN during the attack phase prevents adversarial perturbations from corrupting the running mean and variance statistics. This simple change consistently improves clean accuracy across various architectures without compromising robustness.

Additionally, the choice of activation function and model capacity influences the effectiveness of these tricks. While smooth activations like Softplus significantly benefit smaller models (e.g., ResNet-18), their impact is less pronounced on larger architectures like WideResNet-34-10.

Performance across different model architectures and the impact of Batch Normalization (BN) modes on clean and robust accuracy.

Reclaiming State-of-the-Art

By combining these insights into a standardized "bag of tricks"—including optimized weight decay ($5 \times 10^{-4}$), eval-mode BN for attacks, and label smoothing—we re-evaluated existing frameworks. Applying this recipe to the TRADES method on a WRN-34-20 architecture resulted in an AutoAttack accuracy of 54.39%, reclaiming the top position on the CIFAR-10 leaderboard without the need for additional training data or complex new algorithms.

Conclusion

This research highlights the importance of standardized hyperparameter tuning in the evaluation of adversarial defenses. Many "new" methods may simply be benefiting from better implementation details. We call for the community to adopt more rigorous benchmarking practices, ensuring that future progress in adversarial robustness is driven by genuine algorithmic innovation.

Summary updated

BibTeX
@inproceedings{pang2021bag,
  author = {Tianyu Pang and Xiao Yang and Yinpeng Dong and Hang Su and Jun Zhu},
  title = {{Bag of Tricks for Adversarial Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=Xb8xvrtB8Ce},
}
Figure