Do Wider Neural Networks Really Help Adversarial Robustness?

Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, Quanquan Gu

Introduction

While the deep learning community generally believes that "wider is better" for adversarial robustness, increasing network width harbors a hidden catch: it actually worsens a model's perturbation stability. By identifying this fundamental trade-off and proposing a new "Width Adjusted Regularization" (WAR) method, researchers demonstrate how to properly tune wider models to unleash their full defensive potential.

The Problem with "Wider is Better"

Adversarial training—training a model on examples intentionally designed to fool it—is currently the most effective defense against malicious inputs. Early empirical work suggested that robust models require significantly more capacity (width) than standard models to capture complex decision boundaries.

However, the underlying mechanics of this requirement remained fuzzy. Researchers often fine-tune hyperparameters, such as the regularization strength $\lambda$, on small, fast-to-train models and simply copy those settings over to massive, wide networks. A strange phenomenon emerges under this practice: wider models actually show worse stability during training. This paper investigates the fundamental relationship between a network's width and its ability to remain stable under attack.

Deconstructing Robustness and the WAR Approach

The researchers approach the problem through three distinct lenses:

1. Decomposition of Robustness: They define a new metric called perturbation stability—the fraction of inputs whose predictions remain unchanged within a small neighborhood of perturbations. They argue that "Robust Accuracy" is simply the intersection of "Natural Accuracy" (being correct on clean data) and "Perturbation Stability" (not changing predictions when nudged).

Decomposition of Robustness

2. Theoretical Analysis (NTK): Using Neural Tangent Kernel (NTK) theory, the authors mathematically prove that as a network gets wider ($m$), its local Lipschitz constant (a measure of how rapidly the function's output can change) increases by a factor of $\sqrt{m}$. In plain English: wider networks are naturally more sensitive to small input changes.

3. Width Adjusted Regularization (WAR): To counteract this sensitivity, they propose an algorithm that adaptively adjusts the regularization parameter $\lambda$ during training. Instead of a fixed value, WAR monitors the ratio between natural risk and robust risk, automatically increasing the regularization for wider models to keep them stable.

The Double-Edged Sword of Network Width

A core insight of this work is redefining the standard trade-off in adversarial training. We usually discuss the tension between natural accuracy and robust accuracy. This paper argues the real tension is between natural accuracy and perturbation stability. Robust accuracy is merely the byproduct of how well you balance those two factors.

Accuracy and Stability vs. Width

Increasing width makes a model more expressive, which improves natural accuracy. However, it also makes the model's decision landscape "sharper" and more prone to local fluctuations, thereby harming stability. Consequently, the common practice of using the same $\lambda$ for all model sizes is flawed. Because wider models are naturally less stable, they require a much larger $\lambda$ to "tame" their sensitivity. For a WideResNet-34-10, the optimal $\lambda$ was found to be three times larger than what was used in foundational adversarial training papers like TRADES.

Empirical Validation

The authors validated their findings on standard computer vision benchmarks (CIFAR-10 and CIFAR-100) using popular architectures like WideResNet-34 (varying the "widen factor" from 1 to 10) and DenseNet-BC-40.

  • Performance Boost: By simply using a larger, properly tuned $\lambda$, the authors improved the robust accuracy of WideResNet-34-10 on CIFAR-10 by 2.25% over the standard baseline.
  • Efficiency: The proposed WAR method achieved results comparable to, or better than, exhaustive manual tuning. Crucially, it was roughly 4x faster because it eliminated the need for multiple expensive training runs on large models.
  • State-of-the-Art Defense: When evaluated against AutoAttack (the gold standard for evaluating adversarial defenses), the WAR-trained model outperformed several other high-profile defensive techniques, reaching 54.73% robust accuracy on WRN-34-10.

Limitations and Future Directions

The theoretical portion of the paper relies on the Neural Tangent Kernel (NTK) regime, which assumes "infinitely wide" networks and specific initialization conditions. While the empirical results match the theory well, there remains a slight gap between NTK theory and how finite-width networks behave in practice with modern optimizers.

Future work could explore whether these same principles apply to network depth or other architectural choices, such as attention mechanisms in Transformers. Furthermore, the authors advocate that evaluating "perturbation stability" should become a standard practice for anyone developing new adversarial defenses.

Summary updated

Abstract

Adversarial training is a powerful type of defense against adversarial examples. Previous empirical results suggest that adversarial training requires wider networks for better performances. However, it remains elusive how does neural network width affect model robustness. In this paper, we carefully examine the relationship between network width and model robustness. Specifically, we show that the model robustness is closely related to the tradeoff between natural accuracy and perturbation stability, which is controlled by the robust regularization parameter λ. With the same λ, wider networks can achieve better natural accuracy but worse perturbation stability, leading to a potentially worse overall model robustness. To understand the origin of this phenomenon, we further relate the perturbation stability with the network's local Lipschitzness. By leveraging recent results on neural tangent kernels, we theoretically show that wider networks tend to have worse perturbation stability. Our analyses suggest that: 1) the common strategy of first fine-tuning λ on small networks and then directly use it for wide model training could lead to deteriorated model robustness; 2) one needs to properly enlarge λ to unleash the robustness potential of wider models fully. Finally, we propose a new Width Adjusted Regularization (WAR) method that adaptively enlarges λ on wide models and significantly saves the tuning time.
BibTeX
@inproceedings{wu2021do,
  author = {Boxi Wu and Jinghui Chen and Deng Cai and Xiaofei He and Quanquan Gu},
  title = {{Do Wider Neural Networks Really Help Adversarial Robustness?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure