Neural Architecture Dilation for Adversarial Robustness

Yanxi Li, Zhaohui Yang, Yunhe Wang, Chang Xu

Deep learning models often suffer from a "glass jaw": while they achieve superhuman performance on clean data, they are notoriously vulnerable to adversarial attacks. A slight, imperceptible perturbation to an input image can cause a state-of-the-art Convolutional Neural Network (CNN) to fail catastrophically. Traditionally, the primary defense has been adversarial training, which involves training the model on perturbed examples. However, this often leads to a significant "robustness-accuracy trade-off," where gains in security come at the expense of performance on natural, unperturbed data.

The research paper "Neural Architecture Dilation for Adversarial Robustness" (NADAR) introduces a novel architectural approach to mitigate this trade-off. Instead of focusing solely on training objectives, the authors propose that the physical architecture of a network sets a fundamental bound on its potential for robustness.

The Architecture-Robustness Connection

Most existing research focuses on designing better loss functions (such as TRADES) or data augmentation strategies. NADAR shifts the focus toward the network's capacity. The authors argue that adversarial tasks are inherently more complex than standard classification and require additional model capacity to learn robust features without interfering with standard performance.

Existing robust architectures often require designing models from scratch or significantly increasing their size, which is computationally expensive and often results in an "accuracy tax." NADAR addresses this by surgically adding capacity to a high-performing "backbone" network.

Neural Architecture Dilation (NADAR)

The core of the proposed method is the Neural Architecture Dilation (NADAR) framework. This approach treats the problem as a "correction" task rather than a complete redesign.

The NADAR hybrid architecture, combining a fixed backbone with a searched dilation network.

As shown in the architecture diagram, the system consists of:

  1. The Hybrid Network: A fixed, high-quality backbone network ($f_b$) is augmented with a "dilation network" ($f_d$). The outputs of these two components are aggregated via element-wise summation at various stages.
  2. Differentiable NAS: To determine the optimal structure for the dilation network, the authors employ Neural Architecture Search (NAS). They use a differentiable approach to explore a search space containing operations like separable and dilated convolutions.
  3. Bi-Level Optimization: The search process is framed as a bi-level optimization problem. The inner loop optimizes the weights of the dilation network to minimize adversarial loss, while the outer loop optimizes the architecture parameters ($\alpha$) to minimize validation adversarial loss.

Optimization and Constraints

To ensure the dilation network enhances robustness without degrading standard accuracy, the authors introduce a Standard Performance Constraint. This explicitly enforces that the hybrid model's loss on clean data must be equal to or lower than that of the original backbone.

Furthermore, to maintain efficiency, the objective function includes a FLOPs-awareness penalty, discouraging the search from producing overly complex dilation networks. To manage the high computational cost of combining NAS with adversarial training, the authors utilize FreeAT, a method that recycles gradients to accelerate the generation of adversarial examples during the training loop.

Performance Benchmarks

The researchers evaluated NADAR across several benchmarks, including CIFAR-10, CIFAR-100, and ImageNet. The results demonstrate that NADAR consistently achieves a superior balance between natural accuracy and adversarial resistance compared to traditional defense methods.

Comparison of NADAR against state-of-the-art adversarial training and NAS methods on CIFAR-10.

On the CIFAR-10 dataset, using a WideResNet-32-10 backbone, the NADAR-A variant achieved 86.61% natural accuracy and 52.84% PGD-20 robustness. This represents a significant improvement over standard PGD training, which typically drops natural accuracy to approximately 87% while only reaching ~45% robustness. Compared to popular defenses like TRADES, NADAR provides a much better ratio of robustness gain to accuracy loss.

The method also showed strong generalizability across different backbones, such as ResNet-18 and ResNet-101. On ImageNet, dilating a ResNet-50 improved PGD-20 accuracy from 0% (for an undefended model) to nearly 23%, with minimal impact on natural performance.

Conclusion

NADAR suggests that the path to robust artificial intelligence lies not just in how we train models, but in providing them with the appropriate architectural "muscles" to resist adversarial perturbations. By treating the dilation network as a corrector that provides signals to fix classification errors on adversarial examples, NADAR effectively breaks the traditional robustness-accuracy trade-off. While the search cost remains significant, the framework offers a scalable way to harden existing high-performance models against attack.

Summary updated

BibTeX
@inproceedings{li2021neural,
  author = {Yanxi Li and Zhaohui Yang and Yunhe Wang and Chang Xu},
  title = {{Neural Architecture Dilation for Adversarial Robustness}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=5XrU3V2pxJa},
}
Figure