Plug-And-Pipeline: Efficient Regularization for Single-Step Adversarial Training

Vivek B. S., Ambareesh Revanur, Naveen Venkat, R. Venkatesh BabuWorkshops @ CVPR 2020 · OpenReview

Abstract

Adversarial Training (AT) is a straightforward solution to learn robust models by augmenting the training mini-batches with adversarial samples. Adversarial attack methods range from simple non-iterative (single-step) methods to computationally complex iterative (multi-step) methods. Although the single-step methods are efficient, the models trained using these methods merely appear to be robust, due to the masked gradients. In this work, we propose a novel regularizer named Plug-And-Pipeline (PAP) for single-step AT. The proposed regularizer attenuates the gradient masking effect by promoting the model to learn similar representations for both single-step and multi-step adversaries. Further, we present a novel pipelined approach that allows an efficient implementation of the proposed regularizer. Plug-And-Pipeline yields robustness comparable to multi-step AT methods, while requiring a low computational overhead, similar to that of single-step AT methods.

Introduction

Adversarial Training (AT) remains the primary defense against adversarial perturbations. However, practitioners often face a stark trade-off: fast single-step methods (like FGSM) are computationally efficient but suffer from "gradient masking," where the model learns to fool the specific training attack without gaining true robustness. Conversely, multi-step methods (like PGD) provide genuine robustness but are prohibitively expensive, often requiring 7 to 40 times more computation.

This work introduces Plug-And-Pipeline (PAP), a regularization scheme designed to bridge this gap. PAP enables models to achieve the high-grade robustness of multi-step training while maintaining the efficiency of single-step methods by amortizing the cost of strong adversary generation across training iterations.

The Plug-And-Pipeline Mechanism

The PAP approach is built upon two core innovations: feature consistency regularization and a pipelined generation process.

Feature Consistency

PAP forces the network to produce similar internal representations (pre-softmax activations) for both a "weak" single-step adversary and a "strong" multi-step adversary. This encourages the model to learn a smooth loss landscape, preventing the jagged gradients characteristic of masked models.

The Pipelining Trick

To avoid the high cost of generating multi-step adversaries in every iteration, PAP maintains a small "global memory" of samples (typically 10% of a mini-batch size). In each training step, the algorithm performs only a single step of the attack on these samples. Over $N$ iterations, these samples evolve into strong $N$-step adversaries.

The Plug-And-Pipeline (PAP) mechanism for efficient adversarial sample generation.

By spreading the computational load of a multi-step attack across several iterations, the amortized cost per batch remains low. The authors define a measure $\gamma$-FGSM, representing backprops per sample; while PGD-AT has a $\gamma$ of 7 or 40, PAP achieves a $\gamma$ of just 1.2.

Experimental Evaluation

The method was evaluated on standard benchmarks: MNIST (using a small CNN) and CIFAR-10 (using ResNet-18). The results demonstrate that PAP-trained models are significantly more robust than standard single-step models and nearly as effective as expensive multi-step alternatives.

Key Results

  • Robustness: On CIFAR-10 under a 20-step PGD attack, standard FGSM-AT accuracy drops to 0% due to gradient masking. In contrast, PAP-FGSM maintains ~46.3% accuracy, nearing the ~48.8% achieved by the much slower PGD-AT.
  • Efficiency: PAP requires only a 10% increase in batch size. On MNIST, PAP is approximately 33x faster than standard PGD-AT while maintaining comparable accuracy.
  • Resistance to Gradient Masking: A critical test for true robustness is performance against multi-restart attacks. As shown below, PAP models maintain stable accuracy even as the number of attack restarts increases, confirming that the robustness is genuine and not a result of gradient masking.
Robustness of PAP-trained models against multi-restart PGD attacks on MNIST and CIFAR-10.

Conclusion

Plug-And-Pipeline offers a practical, "plug-and-play" solution for robust deep learning. By combining feature consistency with an efficient pipelined attack generation, it breaks the efficiency-robustness trade-off. While future work may explore scaling to larger datasets like ImageNet and further hyperparameter optimization, PAP currently provides a high-performance defense accessible to researchers with limited computational budgets.

Summary updated

Abstract

Adversarial Training (AT) is a straight forward solution to learn robust models by augmenting the training mini-batches with adversarial samples. Adversarial attack methods range from simple non-iterative (single-step) methods to computationally complex iterative (multi-step) methods. Although the single-step methods are efficient, the models trained using these methods merely appear to be robust, due to the masked gradients. In this work, we propose a novel regularizer named Plug-And-Pipeline (PAP) for single-step AT. The proposed regularizer attenuates the gradient masking effect by promoting the model to learn similar representations for both single-step and multi-step adversaries. Further, we present a novel pipelined approach that allows an efficient implementation of the proposed regularizer. Plug-And-Pipeline yields robustness comparable to multi-step AT methods, while requiring a low computational overhead, similar to that of single-step AT methods.
BibTeX
@inproceedings{s2020plug,
  author = {Vivek B. S. and Ambareesh Revanur and Naveen Venkat and R. Venkatesh Babu},
  title = {{Plug-And-Pipeline: Efficient Regularization for Single-Step Adversarial Training}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=UjD1zKDPatD},
}
Figure