Abstract
The progress in the last decade has enabled machine learning models to achieve impressive performance across a wide range of tasks in Computer Vision. However, a plethora of works have demonstrated the susceptibility of these models to adversarial samples. Adversarial training procedure has been proposed to defend against such adversarial attacks. Adversarial training methods augment mini-batches with adversarial samples, and typically single-step (non-iterative) methods are used for generating these adversarial samples. However, models trained using single-step adversarial training converge to degenerative minima where the model merely appears to be robust. The pseudo robustness of these models is due to the gradient masking effect. Although multi-step adversarial training helps to learn robust models, they are hard to scale due to the use of iterative methods for generating adversarial samples. To address these issues, we propose three different types of regularizers that help to learn robust models using single-step adversarial training methods. The proposed regularizers mitigate the effect of gradient masking by harnessing on properties that differentiate a robust model from that of a pseudo robust model. Performance of models trained using the proposed regularizers is on par with models trained using computationally expensive multi-step adversarial training methods.
Introduction: The Challenge of Adversarial Robustness
Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial attacks—small, intentional perturbations to input data that can drastically change a model's prediction. To defend against these, researchers employ Adversarial Training (AT), a process where the model is trained on adversarial samples.
However, a significant trade-off exists between efficiency and security. Single-step methods, such as the Fast Gradient Sign Method (FGSM), are computationally efficient and scalable but often fall victim to gradient masking. In this state, the model develops a jagged, non-smooth loss surface that "fools" simple attackers during training but remains vulnerable to sophisticated iterative attacks. This phenomenon is known as pseudo-robustness. Conversely, multi-step methods like Projected Gradient Descent (PGD) produce genuine robustness but are 10–40x slower, making them difficult to scale to large datasets like ImageNet.
Proposed Regularization Techniques
To bridge the gap between single-step speed and multi-step security, we propose three novel regularizers (SAT-R1, SAT-R2, and SAT-R3) designed to penalize the mathematical signatures of gradient masking:
- SAT-R1 (Similarity to Iterative): This regularizer penalizes the distance between the model's internal representations (pre-softmax outputs) of a single-step adversarial sample and an iterative (multi-step) sample. To maintain efficiency, the authors demonstrate that generating only one iterative sample per mini-batch is sufficient to guide the entire batch toward a robust state.
- SAT-R2 (Similarity to Random-Start): Similar to R1, this compares the single-step sample to one generated with a random noise start, preventing the model from converging to the degenerate minima that cause gradient masking.
- SAT-R3 (Loss Monotonicity): Truly robust models exhibit a monotonic increase in loss as perturbation size increases. SAT-R3 explicitly penalizes instances where the loss at a small perturbation is higher than or too close to the loss at a large perturbation, forcing a smoother loss surface.
Experimental Analysis and Insights
Evaluations on MNIST, CIFAR-10, and ImageNet-subset show that these regularizers achieve performance on par with the gold-standard PGD training while remaining significantly faster. A key insight driving this work is the observation that in the early stages of training, single-step and multi-step attacks produce very similar results. Gradient masking only emerges as the model "learns" to hide from simple attackers later in the training process.
On CIFAR-10, standard FGSM-AT collapses to near 0% accuracy against PGD attacks. In contrast, our proposed SAT-R1 achieves ~46.8% accuracy, nearly matching the ~47.9% achieved by the much slower PGD-AT. Rigorous sanity tests confirmed that this robustness is genuine and not a result of further gradient masking.
Conclusion
By targeting the geometric properties of the loss surface, the proposed regularizers enable scalable, single-step adversarial training that provides genuine security. While SAT-R3 is the most efficient, SAT-R1 and SAT-R2 provide the highest levels of robustness. Future work may explore optimal training curricula, such as sequential boosting, to further enhance the efficiency and efficacy of these defenses.
Summary updated