Abstract
Adversarial Training (AT) is a straightforward solution to learn robust models by augmenting the training mini-batches with adversarial samples. Adversarial attack methods range from simple non-iterative (single-step) methods to computationally complex iterative (multi-step) methods. Although the single-step methods are efficient, the models trained using these methods merely appear to be robust, due to the masked gradients. In this work, we propose a novel regularizer named Plug-And-Pipeline (PAP) for single-step AT. The proposed regularizer attenuates the gradient masking effect by promoting the model to learn similar representations for both single-step and multi-step adversaries. Further, we present a novel pipelined approach that allows an efficient implementation of the proposed regularizer. Plug-And-Pipeline yields robustness comparable to multi-step AT methods, while requiring a low computational overhead, similar to that of single-step AT methods.
Introduction
Adversarial Training (AT) remains the primary defense against adversarial perturbations. However, practitioners often face a stark trade-off: fast single-step methods (like FGSM) are computationally efficient but suffer from "gradient masking," where the model learns to fool the specific training attack without gaining true robustness. Conversely, multi-step methods (like PGD) provide genuine robustness but are prohibitively expensive, often requiring 7 to 40 times more computation.
This work introduces Plug-And-Pipeline (PAP), a regularization scheme designed to bridge this gap. PAP enables models to achieve the high-grade robustness of multi-step training while maintaining the efficiency of single-step methods by amortizing the cost of strong adversary generation across training iterations.
The Plug-And-Pipeline Mechanism
The PAP approach is built upon two core innovations: feature consistency regularization and a pipelined generation process.
Feature Consistency
PAP forces the network to produce similar internal representations (pre-softmax activations) for both a "weak" single-step adversary and a "strong" multi-step adversary. This encourages the model to learn a smooth loss landscape, preventing the jagged gradients characteristic of masked models.
The Pipelining Trick
To avoid the high cost of generating multi-step adversaries in every iteration, PAP maintains a small "global memory" of samples (typically 10% of a mini-batch size). In each training step, the algorithm performs only a single step of the attack on these samples. Over $N$ iterations, these samples evolve into strong $N$-step adversaries.
By spreading the computational load of a multi-step attack across several iterations, the amortized cost per batch remains low. The authors define a measure $\gamma$-FGSM, representing backprops per sample; while PGD-AT has a $\gamma$ of 7 or 40, PAP achieves a $\gamma$ of just 1.2.
Experimental Evaluation
The method was evaluated on standard benchmarks: MNIST (using a small CNN) and CIFAR-10 (using ResNet-18). The results demonstrate that PAP-trained models are significantly more robust than standard single-step models and nearly as effective as expensive multi-step alternatives.
Key Results
- Robustness: On CIFAR-10 under a 20-step PGD attack, standard FGSM-AT accuracy drops to 0% due to gradient masking. In contrast, PAP-FGSM maintains ~46.3% accuracy, nearing the ~48.8% achieved by the much slower PGD-AT.
- Efficiency: PAP requires only a 10% increase in batch size. On MNIST, PAP is approximately 33x faster than standard PGD-AT while maintaining comparable accuracy.
- Resistance to Gradient Masking: A critical test for true robustness is performance against multi-restart attacks. As shown below, PAP models maintain stable accuracy even as the number of attack restarts increases, confirming that the robustness is genuine and not a result of gradient masking.
Conclusion
Plug-And-Pipeline offers a practical, "plug-and-play" solution for robust deep learning. By combining feature consistency with an efficient pipelined attack generation, it breaks the efficiency-robustness trade-off. While future work may explore scaling to larger datasets like ImageNet and further hyperparameter optimization, PAP currently provides a high-performance defense accessible to researchers with limited computational budgets.
Summary updated