Introduction
Adversarial Training (AT) has emerged as the primary defense against adversarial attacks. A common heuristic in the field is that increasing network capacity—specifically network width—is essential for learning robust decision boundaries. However, recent observations suggest a "width-robustness gap": wider models often exhibit higher natural accuracy but fail to provide proportional gains in robustness, sometimes even performing worse than narrower models under the same training configurations. This paper investigates the underlying causes of this phenomenon and proposes a novel regularization strategy to bridge the gap.
The Conceptual Framework: Perturbation Stability
To analyze this issue, the authors distinguish between Correctly Classified Samples and Stable Examples. A sample is stable if its prediction remains unchanged within an $\epsilon$-ball neighborhood. Robust Samples are the intersection of these two sets: they are both correctly classified and stable.
The core argument is that the fundamental trade-off in adversarial training is not merely between natural and robust accuracy, but between natural accuracy and Perturbation Stability. Robust accuracy is effectively a byproduct of how well these two factors are balanced.
Theoretical Analysis: The Instability of Width
Using the Neural Tangent Kernel (NTK) framework, the authors provide a theoretical explanation for why wider networks are inherently less stable. They prove that the local Lipschitz constant—a measure of a function's sensitivity to input perturbations—scales with the square root of the network width ($\sqrt{m}$).
Consequently, as a network grows wider, it becomes mathematically predisposed to be more sensitive to small input changes. This means that if the regularization strength (the "dampening" factor) is not increased alongside the width, the added capacity of a wide network actually works against its robustness.
Empirical Evidence: The Width-Stability Trade-off
Experiments on CIFAR-10 and CIFAR-100 using WideResNet architectures confirm these theoretical predictions. As the "widen-factor" increases, natural accuracy improves, but perturbation stability consistently declines.
This finding highlights a critical flaw in standard research practices: hyperparameters (like the regularization strength $\lambda$) tuned on small models cannot be directly "copy-pasted" to larger models. Wider models require significantly higher $\lambda$ values to counteract their inherent instability.
Width Adjusted Regularization (WAR)
To address this, the authors introduce Width Adjusted Regularization (WAR). WAR is an adaptive algorithm that dynamically adjusts the regularization parameter $\lambda$ during training. Instead of costly manual grid searches for every model size, WAR maintains a balance between natural accuracy and stability on the fly.
Key Results
The application of WAR leads to significant improvements across standard benchmarks:
- Robustness Boost: On CIFAR-10, increasing $\lambda$ for a WideResNet-34-10 based on the WAR analysis improved robust accuracy by 2.25%.
- Efficiency: WAR achieves state-of-the-art results with a ~4x speedup in the tuning process compared to exhaustive manual search.
- State-of-the-Art Validation: Models trained with WAR achieved a robust accuracy of 54.73% on CIFAR-10 against the AutoAttack ensemble, outperforming standard TRADES and other established defenses.
Conclusion
The assumption that width automatically translates to robustness is incomplete. While wider networks have the capacity for higher robustness, their inherent instability works against them. By recognizing the scaling relationship between width and the Lipschitz constant, and employing adaptive techniques like Width Adjusted Regularization, researchers can finally unlock the full defensive potential of large-scale neural networks.
Summary updated