Abstract
Deep Neural Networks (DNNs) can be easily fooled by Adversarial Examples (AEs) with differences imperceptible to the human eye. To maintain this imperceptibility, existing attacks typically bound adversarial perturbations using the $\ell_\infty$ norm, which serves as the standard for fair comparison. However, this study finds that the $\ell_\infty$ norm alone is insufficient to measure attack strength, particularly regarding transferability—the ability of AEs to fool black-box victim models. Through comprehensive experiments with 7 transfer-based attacks and 13 models, we demonstrate that the $\ell_2$ norm significantly affects transferability in $\ell_\infty$-bounded attacks. We advocate that attack strength should be measured by both $\ell_\infty$ and $\ell_2$ norms to prevent misleading performance gains achieved simply by increasing perturbation energy.
Introduction
In the field of adversarial machine learning, "transferability" is the benchmark for success: can an attack crafted for a known surrogate model successfully deceive an unseen victim model? To ensure these attacks remain hidden, researchers limit the maximum change allowed for any single pixel, known as the $\ell_\infty$ norm. The community has long assumed that a fixed $\ell_\infty$ budget (e.g., 16/255) creates a level playing field. However, this assumption ignores the total "volume" or energy of the perturbation, measured by the $\ell_2$ norm. This paper investigates how this hidden strength inflates transferability results and leads to biased comparisons between algorithms.
The $\ell_2$ Gap in Adversarial Attacks
Iterative attacks like Projected Gradient Descent (PGD) often produce "sparse" perturbations, changing only a few pixels significantly and resulting in a low $\ell_2$ distance (RMSE $\approx$ 7). In contrast, simpler or more aggressive attacks like FGSM push nearly every pixel toward the $\ell_\infty$ limit, resulting in much higher $\ell_2$ distances (RMSE $\approx$ 16). The success of many state-of-the-art transfer attacks is often not due to superior optimization, but simply because they produce perturbations with 70% to 130% larger $\ell_2$ distances.
The "Equalizer" Effect
To isolate the effect of the $\ell_2$ norm, the researchers introduced a modified loss function that allows for explicitly "dialing up" the RMSE while keeping the $\ell_\infty$ bound constant. The most striking finding is the "Equalizer" effect: when different attack algorithms are forced to produce the same $\ell_2$ distance, their transferability becomes almost identical. The specific optimization path matters far less than the total perturbation energy.
Visibility and Perceptibility Trade-offs
Larger $\ell_2$ distances make adversarial examples more perceivable to humans. By ignoring $\ell_2$, researchers may inadvertently favor "noisier" attacks that are easier to spot simply because they transfer better. This highlights the need for multi-norm measurements and the exploration of perceptual metrics like the Structural Similarity Index (SSIM) to ensure defenses are not just overfitting to low-energy attacks.
Conclusion
The $\ell_\infty$ norm is an incomplete metric for evaluating adversarial strength. To ensure truly unbiased and fair evaluations, researchers must measure and report both $\ell_\infty$ and $\ell_2$ distances. This shift is necessary to prevent the community from being misled by "tricky" enhancements that rely on increased perturbation energy rather than algorithmic innovation.
Summary updated