Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks

Hanxun Huang, Yisen Wang, Sarah Monazam Erfani, Quanquan Gu, James Bailey, Xingjun Ma

Introduction

Deep Neural Networks (DNNs) have achieved remarkable success in various tasks but remain notoriously fragile to adversarial attacks—small, often imperceptible perturbations that can drastically change a model's prediction. While Adversarial Training (AT) is currently the most effective defense, research has largely focused on the training process itself rather than the architectural design of the networks. The prevailing "bigger is better" mantra suggests that massive models with high capacity are necessary to absorb the complexity of adversarial examples. However, this paper challenges that assumption, investigating the specific "architectural ingredients" that contribute to adversarial stability.

Methods

The researchers conducted an extensive empirical study using WideResNet-34-10 (WRN-34-10) as a baseline. They decomposed the network into three distinct stages and performed a fine-grained grid search, varying the depth (number of residual blocks) and width (number of filters) within each stage to observe the impact on robustness.

WRN-34-10 architecture and grid search results on depth.

To ground their empirical findings, the authors introduced a theoretical framework based on the Lipschitz constant. In the context of neural networks, the Lipschitz constant measures the maximum rate of change of the output relative to the input. A lower constant implies a "smoother" function that is less sensitive to small input perturbations. The authors proved that the upper bound of this constant grows exponentially with depth and is significantly influenced by width, providing a mathematical basis for why excessive capacity can lead to "jittery" decision boundaries and reduced robustness.

Key Insights: The "Last Stage" Secret

The most significant discovery of the study is the counter-intuitive relationship between capacity and robustness in the final stage of a network. While earlier stages require sufficient capacity to extract complex features, the deepest layers benefit from being constrained.

Stage-wise analysis of depth on PGD robustness.

As shown in the stage-wise analysis, reducing the depth or width in the final group of residual blocks consistently improves adversarial robustness. This suggests a Capacity vs. Stability Trade-off: if a model is too small, it lacks the representational power to learn; if it is too large in the final stages, its Lipschitz constant explodes, making it easier to fool. The authors found that an optimally configured network—one with more blocks in the early stages and fewer in the last—significantly outperforms standard, balanced architectures under the same parameter budget.

Empirical Results

Based on these insights, the authors proposed WRN-34-R, a reconfigured version of the standard WideResNet. The results across multiple benchmarks were compelling:

  • CIFAR-10/100: WRN-34-R consistently outperformed the standard WRN-34-12 by 1% to 3% across various AT methods (SAT, TRADES, MART), despite having a more efficient parameter distribution.
  • Efficiency: The optimized configuration achieved approximately 54% robustness with only 17M parameters, whereas a standard configuration required 22M parameters to reach the same performance level.
  • ImageNet: Applying the "reduction rule" to ResNet-50 improved PGD10 robustness from 30.48% to 31.14% while simultaneously increasing clean accuracy, proving the rule scales to large-scale datasets.
  • Universality: The principles were found to be architecture-agnostic, improving the robustness of VGG, DenseNet, and even architectures discovered via Neural Architecture Search (DARTS).

Conclusion and Future Work

This research demonstrates that architectural configuration is as critical as the training algorithm for building robust DNNs. By strategically reducing capacity in the final stages of a network, researchers can lower the Lipschitz constant and create more stable models without sacrificing clean performance.

While these findings provide a powerful heuristic for manual design, future work remains in automating these insights. Integrating Lipschitz-based constraints directly into Neural Architecture Search (NAS) could lead to the discovery of even more robust structures. Additionally, closing the persistent gap between clean and adversarial accuracy remains a fundamental challenge for the field.

Summary updated

BibTeX
@inproceedings{huang2021exploring,
  author = {Hanxun Huang and Yisen Wang and Sarah Monazam Erfani and Quanquan Gu and James Bailey and Xingjun Ma},
  title = {{Exploring Architectural Ingredients of Adversarially Robust Deep Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure