MAD: Meta Adversarial Defense Benchmark

Xin Peng, Dan Zhou, Guangling Sun, Jie Shi, Li WuarXiv 2023 · DOI · arXiv

Abstract Adversarial training (AT) is a prominent technique employed by deep learning models to defend against adversarial attacks. However, existing AT-based methods suffer from high computational costs, low generalization to unseen attacks, and a trade-off between robustness and clean-sample classification accuracy (CCA). We propose the Meta Adversarial Defense (MAD) benchmark, comprising two large-scale datasets (MAD-M and MAD-C) generated using 30 types of attacks, and an evaluation protocol. We introduce a meta-learning based adversarial training (Meta-AT) algorithm as a baseline. Meta-AT treats defense as a few-shot learning problem, enabling high robustness to unseen attacks with minimal adaptation time. Experimental results show Meta-AT outperforms state-of-the-art methods while maintaining high CCA, addressing the core limitations of traditional AT.

Introduction

Adversarial Training (AT) remains the primary defense against malicious perturbations. Despite its effectiveness, it faces three critical challenges: high computational overhead during training, poor generalization to attacks not seen during training, and a significant drop in accuracy on clean data. The MAD benchmark shifts the focus from building static defenses to developing models capable of rapid adaptation.

Conceptual comparison between traditional Adversarial Training (a) and the proposed Meta-Adversarial Training (d).

Methods: Meta-Adversarial Training (Meta-AT)

The Meta-AT algorithm redefines adversarial defense as a meta-learning task. Instead of training on a static set of adversarial examples, the model learns to adapt its parameters quickly when encountering new attack patterns.

Task-Based Training

The process follows an "N-way, K-shot" framework common in meta-learning. During training, the model is exposed to a diverse library of attacks and learns to update its parameters so that it can adapt to any of them with just a few examples.

Dual Updates

As shown in the workflow below, the algorithm performs "inner" updates to specialize for a specific attack and "outer" updates to optimize the base model's generalizability. This structure allows the model to bridge the gap between different attack domains.

The Meta-AT workflow, illustrating the training and testing phases using attack libraries and ResNet-18.

Evaluation Metric

The authors introduce the Equilibrium Defense Success Rate (EDSR). Standard metrics only look at defense success; EDSR factors in Operating Time (OT). A defense that takes hours to adapt is less valuable than one that takes seconds. The formula $EDSR = DSR \times e^{-OT}$ penalizes slow adaptation.

The MAD Benchmark Datasets

To support this research, the authors developed two comprehensive datasets:

  • MAD-M: Based on MNIST (handwritten digits), featuring 22 different types of attacks.
  • MAD-C: Based on CIFAR-10 (common objects), featuring 29 different types of attacks.
  • The Attack Library: Includes 30 mainstream attacks, such as PGD, DeepFool, FGSM, and black-box attacks like Square and Zoo-Attack. This variety ensures models are tested against a wide spectrum of adversarial strategies.

Key Results

Meta-AT demonstrates significant improvements across all metrics:

  • Superior Robustness: Meta-AT achieved a maximum EDSR of 99.77% on MAD-M and 92.98% on MAD-C, significantly outperforming other state-of-the-art (SOTA) methods like ATTA and YOPO.
  • Minimal Accuracy Loss: On CIFAR-10, the clean accuracy only dropped by 6.24%, whereas other robust training methods often see much steeper declines.
  • Speed: Meta-AT can adapt to a new attack in roughly 0.33 minutes for MNIST and 2 minutes for CIFAR-10, making it viable for real-world applications where threats evolve rapidly.

Conclusion

The MAD benchmark and Meta-AT algorithm provide a new paradigm for adversarial defense. By viewing defense as an agile adaptation process rather than a static shield, Meta-AT overcomes the traditional trade-offs of computational cost and generalization. Future work will focus on reducing dataset size and extending this framework to complex vision tasks like object detection and semantic segmentation.

Summary updated

Abstract

Adversarial training (AT) is a prominent technique employed by deep learning models to defend against adversarial attacks, and to some extent, enhance model robustness. However, there are three main drawbacks of the existing AT-based defense methods: expensive computational cost, low generalization ability, and the dilemma between the original model and the defense model. To this end, we propose a novel benchmark called meta adversarial defense (MAD). The MAD benchmark consists of two MAD datasets, along with a MAD evaluation protocol. The two large-scale MAD datasets were generated through experiments using 30 kinds of attacks on MNIST and CIFAR-10 datasets. In addition, we introduce a meta-learning based adversarial training (Meta-AT) algorithm as the baseline, which features high robustness to unseen adversarial attacks through few-shot learning. Experimental results demonstrate the effectiveness of our Meta-AT algorithm compared to the state-of-the-art methods. Furthermore, the model after Meta-AT maintains a relatively high clean-samples classification accuracy (CCA). It is worth noting that Meta-AT addresses all three aforementioned limitations, leading to substantial improvements. This benchmark ultimately achieved breakthroughs in investigating the transferability of adversarial defense methods to new attacks and the ability to learn from a limited number of adversarial examples. Our codes and attacked datasets address will be available at https://github.com/PXX1110/Meta_AT.
BibTeX
@misc{peng2023mad,
  author = {Xin Peng and Dan Zhou and Guangling Sun and Jie Shi and Li Wu},
  title = {{MAD: Meta Adversarial Defense Benchmark}},
  year = {2023},
  eprint = {2309.09776},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.09776},
}
Figure