Does Network Width Really Help Adversarial Robustness?

Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, Quanquan Gu

Introduction

Adversarial Training (AT) has emerged as the primary defense against adversarial attacks. A common heuristic in the field is that increasing network capacity—specifically network width—is essential for learning robust decision boundaries. However, recent observations suggest a "width-robustness gap": wider models often exhibit higher natural accuracy but fail to provide proportional gains in robustness, sometimes even performing worse than narrower models under the same training configurations. This paper investigates the underlying causes of this phenomenon and proposes a novel regularization strategy to bridge the gap.

The Conceptual Framework: Perturbation Stability

To analyze this issue, the authors distinguish between Correctly Classified Samples and Stable Examples. A sample is stable if its prediction remains unchanged within an $\epsilon$-ball neighborhood. Robust Samples are the intersection of these two sets: they are both correctly classified and stable.

Figure 2: Conceptual relationship between stable examples, correctly classified samples, and robust samples.

The core argument is that the fundamental trade-off in adversarial training is not merely between natural and robust accuracy, but between natural accuracy and Perturbation Stability. Robust accuracy is effectively a byproduct of how well these two factors are balanced.

Theoretical Analysis: The Instability of Width

Using the Neural Tangent Kernel (NTK) framework, the authors provide a theoretical explanation for why wider networks are inherently less stable. They prove that the local Lipschitz constant—a measure of a function's sensitivity to input perturbations—scales with the square root of the network width ($\sqrt{m}$).

Consequently, as a network grows wider, it becomes mathematically predisposed to be more sensitive to small input changes. This means that if the regularization strength (the "dampening" factor) is not increased alongside the width, the added capacity of a wide network actually works against its robustness.

Empirical Evidence: The Width-Stability Trade-off

Experiments on CIFAR-10 and CIFAR-100 using WideResNet architectures confirm these theoretical predictions. As the "widen-factor" increases, natural accuracy improves, but perturbation stability consistently declines.

Figure 6: Metrics against network width. While natural and robust accuracy generally increase with width, perturbation stability significantly decreases.

This finding highlights a critical flaw in standard research practices: hyperparameters (like the regularization strength $\lambda$) tuned on small models cannot be directly "copy-pasted" to larger models. Wider models require significantly higher $\lambda$ values to counteract their inherent instability.

Width Adjusted Regularization (WAR)

To address this, the authors introduce Width Adjusted Regularization (WAR). WAR is an adaptive algorithm that dynamically adjusts the regularization parameter $\lambda$ during training. Instead of costly manual grid searches for every model size, WAR maintains a balance between natural accuracy and stability on the fly.

Key Results

The application of WAR leads to significant improvements across standard benchmarks:

  • Robustness Boost: On CIFAR-10, increasing $\lambda$ for a WideResNet-34-10 based on the WAR analysis improved robust accuracy by 2.25%.
  • Efficiency: WAR achieves state-of-the-art results with a ~4x speedup in the tuning process compared to exhaustive manual search.
  • State-of-the-Art Validation: Models trained with WAR achieved a robust accuracy of 54.73% on CIFAR-10 against the AutoAttack ensemble, outperforming standard TRADES and other established defenses.

Conclusion

The assumption that width automatically translates to robustness is incomplete. While wider networks have the capacity for higher robustness, their inherent instability works against them. By recognizing the scaling relationship between width and the Lipschitz constant, and employing adaptive techniques like Width Adjusted Regularization, researchers can finally unlock the full defensive potential of large-scale neural networks.

Summary updated

BibTeX
@inproceedings{wu2021does,
  author = {Boxi Wu and Jinghui Chen and Deng Cai and Xiaofei He and Quanquan Gu},
  title = {{Does Network Width Really Help Adversarial Robustness?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure