Abstract
Adversarial robustness evaluation is critical for deploying secure deep learning models in real-world environments. Current evaluation methods face a difficult trade-off between the computational efficiency of simple attacks like PGD and the reliability of expensive ensembles like AutoAttack. We introduce the Probability Margin Attack (PMA), a powerful individual white-box attack that operates in probability space rather than logit space. PMA achieves evaluation performance comparable to the gold-standard AutoAttack at approximately 25% of the computational cost. Furthermore, we introduce CC1M, a new million-scale benchmark dataset. Our large-scale evaluation reveals a significant "robustness gap": models that appear secure on standard datasets like ImageNet-1k show significantly higher vulnerability when tested at the million-image scale.
Introduction
Deep learning models are notoriously vulnerable to adversarial examples—carefully crafted, human-imperceptible perturbations that induce misclassification. To build safer AI, researchers must "stress test" models using white-box attacks. However, a dilemma exists in current practice. Standard individual attacks, such as Projected Gradient Descent (PGD), are fast but often fail against models employing "obfuscated gradients." Conversely, ensemble attacks like AutoAttack are highly effective but computationally prohibitive for large-scale evaluation. This research bridges this gap by introducing PMA, a single, efficient, and high-strength attack capable of rigorous million-scale evaluation.
Methods: Probability Margin Attack
The core innovation of this work is the Probability Margin Loss ($\mathcal{L}_{pm}$). While most adversarial attacks calculate the "margin"—the gap between the correct class and the next best guess—using raw logits, PMA operates in the probability space after the softmax layer.
By utilizing probabilities, the loss function naturally incorporates information from all classes via the softmax denominator. The authors prove mathematically that this probability-based loss acts as a weighted combination of targeted and untargeted attacks, allowing the attack to adaptively focus on the most vulnerable class directions.
The attack implementation follows a two-stage strategy inspired by Margin Decomposition (MD). In the first stage, it explores adversarial directions by alternating between maximizing the best-guess probability and minimizing the correct-class probability. In the second stage, it applies the full probability margin loss to fine-tune the perturbation. For maximum evaluation strength, the authors also propose PMA+, an ensemble that combines PMA with a multi-targeted version of Auto-PGD.
Million-Scale Evaluation on CC1M
To move beyond standard benchmarks like ImageNet-1k (which contains 50,000 test images), the authors curated CC1M, a million-scale dataset derived from Conceptual Captions (CC3M). To ensure a high-quality evaluation set, the authors used Local Intrinsic Dimensionality (LID)—a metric measuring data complexity—to filter out "junk" images, broken URLs, and anomalies. This provides a diverse and challenging environment for testing model robustness at scale.
Results and the "Robustness Gap"
PMA consistently outperformed existing individual attacks, such as DLR and Margin Loss, across 29 different state-of-the-art models on the RobustBench leaderboard. The PMA+ ensemble achieved results comparable to AutoAttack (which typically uses 4,900 steps) while requiring only about 25% of the execution time. On several CIFAR-10 models, PMA+ even surpassed AutoAttack, identifying vulnerabilities previously missed by the standard benchmark.
The most striking finding of the study is the "robustness gap" observed during million-scale evaluation. Models that demonstrated over 59% relative robustness on the standard ImageNet-1k test set saw their performance plummet to below 19% when evaluated on CC1M.
This discovery suggests that current robustness evaluations on smaller datasets may be overly optimistic. Large-scale testing is essential for understanding the true safety profile of models intended for real-world, safety-critical applications.
Conclusion
The Probability Margin Attack (PMA) establishes a new standard for efficient and high-strength adversarial evaluation. By bridging the performance gap between individual attacks and expensive ensembles, PMA enables the rigorous testing of models at the million-image scale. The revelation of the robustness gap on CC1M underscores the necessity of large-scale benchmarks for developing truly resilient AI systems. Future research directions include extending these probability-based methods to black-box settings and utilizing Probability Margin Loss during adversarial training to create models with inherent resistance to these stronger attacks.
Summary updated