Introduction
While the deep learning community generally believes that "wider is better" for adversarial robustness, increasing network width harbors a hidden catch: it actually worsens a model's perturbation stability. By identifying this fundamental trade-off and proposing a new "Width Adjusted Regularization" (WAR) method, researchers demonstrate how to properly tune wider models to unleash their full defensive potential.
The Problem with "Wider is Better"
Adversarial training—training a model on examples intentionally designed to fool it—is currently the most effective defense against malicious inputs. Early empirical work suggested that robust models require significantly more capacity (width) than standard models to capture complex decision boundaries.
However, the underlying mechanics of this requirement remained fuzzy. Researchers often fine-tune hyperparameters, such as the regularization strength $\lambda$, on small, fast-to-train models and simply copy those settings over to massive, wide networks. A strange phenomenon emerges under this practice: wider models actually show worse stability during training. This paper investigates the fundamental relationship between a network's width and its ability to remain stable under attack.
Deconstructing Robustness and the WAR Approach
The researchers approach the problem through three distinct lenses:
1. Decomposition of Robustness: They define a new metric called perturbation stability—the fraction of inputs whose predictions remain unchanged within a small neighborhood of perturbations. They argue that "Robust Accuracy" is simply the intersection of "Natural Accuracy" (being correct on clean data) and "Perturbation Stability" (not changing predictions when nudged).
2. Theoretical Analysis (NTK): Using Neural Tangent Kernel (NTK) theory, the authors mathematically prove that as a network gets wider ($m$), its local Lipschitz constant (a measure of how rapidly the function's output can change) increases by a factor of $\sqrt{m}$. In plain English: wider networks are naturally more sensitive to small input changes.
3. Width Adjusted Regularization (WAR): To counteract this sensitivity, they propose an algorithm that adaptively adjusts the regularization parameter $\lambda$ during training. Instead of a fixed value, WAR monitors the ratio between natural risk and robust risk, automatically increasing the regularization for wider models to keep them stable.
The Double-Edged Sword of Network Width
A core insight of this work is redefining the standard trade-off in adversarial training. We usually discuss the tension between natural accuracy and robust accuracy. This paper argues the real tension is between natural accuracy and perturbation stability. Robust accuracy is merely the byproduct of how well you balance those two factors.
Increasing width makes a model more expressive, which improves natural accuracy. However, it also makes the model's decision landscape "sharper" and more prone to local fluctuations, thereby harming stability. Consequently, the common practice of using the same $\lambda$ for all model sizes is flawed. Because wider models are naturally less stable, they require a much larger $\lambda$ to "tame" their sensitivity. For a WideResNet-34-10, the optimal $\lambda$ was found to be three times larger than what was used in foundational adversarial training papers like TRADES.
Empirical Validation
The authors validated their findings on standard computer vision benchmarks (CIFAR-10 and CIFAR-100) using popular architectures like WideResNet-34 (varying the "widen factor" from 1 to 10) and DenseNet-BC-40.
- Performance Boost: By simply using a larger, properly tuned $\lambda$, the authors improved the robust accuracy of WideResNet-34-10 on CIFAR-10 by 2.25% over the standard baseline.
- Efficiency: The proposed WAR method achieved results comparable to, or better than, exhaustive manual tuning. Crucially, it was roughly 4x faster because it eliminated the need for multiple expensive training runs on large models.
- State-of-the-Art Defense: When evaluated against AutoAttack (the gold standard for evaluating adversarial defenses), the WAR-trained model outperformed several other high-profile defensive techniques, reaching 54.73% robust accuracy on WRN-34-10.
Limitations and Future Directions
The theoretical portion of the paper relies on the Neural Tangent Kernel (NTK) regime, which assumes "infinitely wide" networks and specific initialization conditions. While the empirical results match the theory well, there remains a slight gap between NTK theory and how finite-width networks behave in practice with modern optimizers.
Future work could explore whether these same principles apply to network depth or other architectural choices, such as attention mechanisms in Transformers. Furthermore, the authors advocate that evaluating "perturbation stability" should become a standard practice for anyone developing new adversarial defenses.
Summary updated