Abstract: Deep neural networks could be fooled by adversarial examples with trivial differences to original samples. To keep the difference imperceptible in human eyes, researchers bound the adversarial perturbations by the $\ell_\infty$ norm, which is now commonly served as the standard to align the strength of different attacks for a fair comparison. However, we propose that using the $\ell_\infty$ norm alone is not sufficient in measuring the attack strength, because even with a fixed $\ell_\infty$ distance, the $\ell_2$ distance also greatly affects the attack transferability between models. Through the discovery, we reach more in-depth understandings towards the attack mechanism, i.e., several existing methods attack black-box models better partly because they craft perturbations with 70% to 130% larger $\ell_2$ distances. Since larger perturbations naturally lead to better transferability, we thereby advocate that the strength of attacks should be simultaneously measured by both the $\ell_\infty$ and $\ell_2$ norm. Our proposal is firmly supported by extensive experiments on ImageNet dataset from 7 attacks, 4 white-box models, and 9 black-box models.
Introduction
In adversarial machine learning, the goal is to craft "adversarial examples" (AEs)—inputs modified to fool a model while remaining visually indistinguishable from the original to humans. A critical metric for these attacks is transferability: the ability of an attack generated against a known "surrogate" model to successfully fool an unknown "victim" model.
To ensure fair comparisons, the research community typically uses the $\ell_\infty$ norm as a budget constraint, limiting the maximum change allowed for any individual pixel. However, this research demonstrates that the $\ell_\infty$ norm is an incomplete metric. An attack can stay within the $\ell_\infty$ budget while varying significantly in its total perturbation volume, measured by the $\ell_2$ norm. This paper investigates whether the perceived superiority of certain attack algorithms is simply a result of them utilizing more of the available perturbation space.
The $\ell_\infty$ Fallacy and the $\ell_2$ Hypothesis
The core argument is that the $\ell_\infty$ norm acts as a "box" constraint, but it does not account for how much of that box is filled. An attack that changes every pixel by the maximum allowed amount has the same $\ell_\infty$ value as an attack that changes only one pixel, yet the former is significantly more potent and visible. The researchers hypothesized that many "state-of-the-art" transfer-based attacks achieve higher success rates primarily because they produce perturbations with 70% to 130% larger $\ell_2$ distances than baseline methods.
Experimental Methodology
The study utilized the ImageNet validation set and tested seven transfer-based attacks, including standard Gradient Descent (GD), Adam, AdaBelief, and Nesterov-accelerated gradients. These were tested across four white-box surrogate models and nine black-box victim models.
To isolate the effect of the $\ell_2$ norm, the researchers introduced a modified loss function: $L' = L + \gamma\|\mathbf{x} - \mathbf{x}_{org}\|_2$
This allowed for the artificial "boosting" of the $\ell_2$ distance while strictly adhering to the $\ell_\infty$ constraint. Additionally, "Fixed-RMSE" experiments were conducted where hyper-parameters were tuned to ensure all attack methods reached a uniform Root Mean Square Error (RMSE), a proxy for the $\ell_2$ norm.
Key Findings
The Convergence of Transferability
When different attack optimizers are forced to maintain the same $\ell_2$ distance (RMSE), their performance differences largely disappear. As shown in Figure 1, when the RMSE is fixed at approximately 15, the Attack Success Rate (ASR) across various victim models becomes remarkably consistent, regardless of the optimizer used. This suggests that the choice of optimizer (e.g., Adam vs. SGD) is secondary to the total volume of the perturbation.
Boosting Success via $\ell_2$ Distance
The research found a near-perfect correlation between an attack's RMSE and its ASR on unseen models. By utilizing the $\ell_2$ penalty term in the loss function, the researchers could significantly boost the transferability of any given attack method. Figure 2 illustrates this effect: the dashed lines (representing boosted $\ell_2$ versions) show a clear increase in ASR compared to their standard counterparts, even though both remain within the same $\ell_\infty$ bounds.
Conclusion
The findings indicate that the current standard of using only the $\ell_\infty$ norm to evaluate adversarial attacks is insufficient and potentially misleading. Attacks that appear "stronger" are often just more "perceivable" because they involve denser perturbations. To ensure truly fair comparisons and better understand model vulnerabilities, the research community should adopt a dual-norm measurement approach, considering both $\ell_\infty$ and $\ell_2$ distances. Future work may further refine this by exploring perceptual metrics like SSIM to better align adversarial evaluation with human visual perception.
Summary updated