Robust$\mathcal{ART}$: Benchmarking Robustness on Architecture Design and Training Techniques

Shiyu Tang, Ruihao Gong, Wang Yan, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, Philip H. S. Torr, Dacheng Tao2021

TL;DR

This paper introduces RobustART, the first comprehensive benchmark for evaluating the robustness of deep neural networks against adversarial, natural, and system noises. It systematically analyzes how different neural network architectures and training techniques influence robustness, providing crucial insights for building more resilient AI systems.

What Problem Does This Paper Address?

Deep neural networks (DNNs) have achieved remarkable success, but they remain vulnerable to various forms of noise, including adversarial perturbations, natural corruptions, and system-level variations. While existing research has focused on developing defenses against adversarial attacks, there's a significant gap in understanding how fundamental design choices, like network architecture and training methodologies, impact overall robustness. This lack of comprehensive benchmarking hinders the development of truly robust DNNs, especially for safety-critical applications.

Methods

The authors introduce RobustART, a comprehensive benchmark and open-source platform for evaluating DNN robustness. Key aspects of their approach include:

  • Extensive Architecture Coverage: They evaluate 49 human-designed off-the-shelf architectures (including CNNs, Transformers, and MLP-Mixers) and over 1200 architectures generated by Neural Architecture Search (NAS).
  • Diverse Noise Types: The benchmark considers adversarial noises (from 9 different attacks with varying magnitudes), natural noises (ImageNet-C, -P, -A, -O), and system noises (ImageNet-S).
  • Systematic Training Technique Analysis: They investigate the impact of over 10 mainstream training techniques (e.g., adversarial training, data augmentation, weight averaging, label smoothing).
  • Fair Comparisons: A core principle is to conduct fair comparisons by aligning training settings for architectures and evaluating techniques across diverse architectures.
  • Open-Source Framework: They provide an open-source platform with a model zoo, toolkit, and datasets to facilitate community research.

Datasets

  • ImageNet-1K: Used for training and evaluating models.
  • ImageNet-C, ImageNet-P, ImageNet-A, ImageNet-O: Datasets representing natural noise corruptions.
  • ImageNet-S: Dataset for evaluating robustness against system noises.

Key Insights and Ideas

  • Adversarial Training is a Game-Changer for Transformers and MLP-Mixers: Unlike CNNs, adversarial training universally improves the robustness of Transformers and MLP-Mixers against all noise types (adversarial, natural, and system) with minimal clean accuracy drop.
  • Architecture Matters Most: Model architecture is a more critical factor for robustness than model size.
  • Architecture Strengths Vary by Noise Type: CNNs (e.g., ResNet) generally excel at natural and system noise robustness, while Transformers (e.g., ViT) are more robust against adversarial noises.
  • Noise Diversity is Crucial: Different noise types can lead to contradictory observations about model robustness, highlighting the need for comprehensive evaluation.
  • Lightweight Models Have Limits: For some lightweight architectures, simply increasing model size or data doesn't necessarily improve robustness.

Key Results

  • Top Architectures:
    • Adversarial Robustness: ViT, VITAE, and DeiT rank highest.
  • Natural Noise Robustness: ResNeXt, WideResNet, and ResNet perform best.
  • System Noise Robustness: EfficientNet, DenseNet, and ResNeXt lead.
  • Training Technique Impacts: Adversarial training significantly boosts robustness for Transformers and MLP-Mixers. Data augmentation can improve natural noise robustness but may decrease adversarial robustness.
  • Model Size vs. Robustness: For many model families (like ResNets), increasing size improves robustness. However, this trend doesn't hold for all lightweight models.

Limitations and Future Work

  • Evolving Landscape: The rapid emergence of new architectures and techniques means the benchmark will need continuous updates.
  • Task Focus: The current version focuses on image classification; future work will expand to object detection and semantic segmentation.
  • Deeper Analysis: While intriguing phenomena are observed, further in-depth studies are needed to fully understand their causes.
  • Community Contribution: The authors emphasize the importance of community contributions to expand the benchmark and its ecosystem.

Summary updated

BibTeX
@misc{tang2021robust,
  author = {Shiyu Tang and Ruihao Gong and Wang Yan and Aishan Liu and Jiakai Wang and Xinyun Chen and Fengwei Yu and Xianglong Liu and Dawn Song and Alan Yuille and Philip H. S. Torr and Dacheng Tao},
  title = {{Robust$\textbackslash\{\}mathcal\{ART\}$: Benchmarking Robustness on Architecture Design and Training Techniques}},
  year = {2021},
}
Figure