Adversarial Initialization with Universal Adversarial Perturbation: A New Approach to Fast Adversarial Training

Chao Pan, Qing Li, Xin YaoAAAI 2024 · DOI

Introduction

Deep Learning models are notoriously vulnerable to adversarial attacks—small, intentional perturbations to input data that cause misclassification. Adversarial Training (AT) remains the primary defense mechanism, involving the inclusion of adversarial examples in the training loop. However, standard AT is computationally expensive, often requiring multiple steps to generate strong attacks for every training batch.

Fast Adversarial Training (FAT) attempts to mitigate this by using single-step attacks like the Fast Gradient Sign Method (FGSM). While efficient, FAT is prone to Catastrophic Overfitting (CO), a phenomenon where the model develops robustness only against the specific weak attack used during training, leaving it vulnerable to stronger multi-step attacks. Recent solutions like FGSM-PGI have addressed CO by "recycling" perturbations from previous epochs, but these methods require storing a unique perturbation for every image in the dataset, creating a massive memory bottleneck for large-scale applications.

The Universal Adversarial Perturbation (UAP) Insight

This research identifies a critical observation regarding data augmentation: because images are frequently flipped, cropped, or rotated during training, a perturbation recycled for a specific image across different epochs effectively acts as a Universal Adversarial Perturbation (UAP) for that image and all its augmented variants. This insight suggests that storing per-sample noise is unnecessary to prevent catastrophic overfitting.

Figure 1: Accuracy on training data comparing original and augmented images across different attack types. Note how FGSM-PGI maintains consistent accuracy across both, suggesting the recycled perturbations generalize across augmentations.

Proposed Methodology

Based on the UAP insight, the authors propose three strategies to initialize adversarial training without the need for per-sample storage:

  1. Single UAP: A single perturbation vector is shared across the entire dataset.
  2. Class-based UAPs: One perturbation vector is maintained for each class (e.g., separate vectors for "dogs" vs. "cats").
  3. Feature-based UAPs: The most advanced method, which uses a small, two-layer Multilayer Perceptron (MLP) to map image features to a specific UAP from a small, pre-defined pool.

By shifting from per-sample memory to shared UAPs, the model learns to defend against common adversarial patterns rather than memorizing pixel-specific noise. This approach exploits the fact that similar images likely share adversarial vulnerabilities.

Experimental Results

The proposed methods were evaluated on standard benchmarks including CIFAR-10, CIFAR-100, and Tiny ImageNet. The results demonstrate that UAP-based initialization achieves state-of-the-art (SOTA) robustness while significantly reducing memory overhead.

Figure 6: Comparison of the proposed UAP methods against existing adversarial training techniques. The Feature-based UAP method achieves superior AutoAttack (AA) accuracy with a drastically lower memory footprint compared to FGSM-PGI.

Key Performance Metrics:

  • Robustness: On CIFAR-10, the Feature-based FGSM-UAP achieved an AutoAttack (AA) accuracy of 49.18%, outperforming both FGSM-PGI (48.76%) and traditional PGD-AT (48.93%).
  • Efficiency: The proposed methods are approximately 3.6 times faster than traditional PGD-based adversarial training.
  • Memory Footprint: While FGSM-PGI requires storing perturbations for the "Whole dataset," the "Single UAP" method requires only one vector. Even the "Feature-based" version requires only 50–300 vectors, regardless of dataset size.
  • Stability: The methods successfully prevented Catastrophic Overfitting across all tested datasets, maintaining high robust accuracy throughout the training process.

Conclusion

By leveraging the relationship between data augmentation and universal perturbations, this work provides a memory-efficient and high-performance framework for Fast Adversarial Training. The transition from per-sample storage to feature-based UAPs allows for SOTA robustness with negligible memory costs, paving the way for scalable adversarial defenses on massive datasets like ImageNet-1k.

Summary updated

BibTeX
@inproceedings{pan2024adversarial,
  author = {Chao Pan and Qing Li and Xin Yao},
  title = {{Adversarial Initialization with Universal Adversarial Perturbation: A New Approach to Fast Adversarial Training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i19.30147},
}
Figure