Measuring $\ell_\infty$ Attacks by the $\ell_2$ Norm

Sizhe Chen, Qinghua Tao, Zhixing Ye, Xiaolin Huang

Abstract

Deep Neural Networks (DNNs) can be easily fooled by Adversarial Examples (AEs) with differences imperceptible to the human eye. To maintain this imperceptibility, existing attacks typically bound adversarial perturbations using the $\ell_\infty$ norm, which serves as the standard for fair comparison. However, this study finds that the $\ell_\infty$ norm alone is insufficient to measure attack strength, particularly regarding transferability—the ability of AEs to fool black-box victim models. Through comprehensive experiments with 7 transfer-based attacks and 13 models, we demonstrate that the $\ell_2$ norm significantly affects transferability in $\ell_\infty$-bounded attacks. We advocate that attack strength should be measured by both $\ell_\infty$ and $\ell_2$ norms to prevent misleading performance gains achieved simply by increasing perturbation energy.

Introduction

In the field of adversarial machine learning, "transferability" is the benchmark for success: can an attack crafted for a known surrogate model successfully deceive an unseen victim model? To ensure these attacks remain hidden, researchers limit the maximum change allowed for any single pixel, known as the $\ell_\infty$ norm. The community has long assumed that a fixed $\ell_\infty$ budget (e.g., 16/255) creates a level playing field. However, this assumption ignores the total "volume" or energy of the perturbation, measured by the $\ell_2$ norm. This paper investigates how this hidden strength inflates transferability results and leads to biased comparisons between algorithms.

The $\ell_2$ Gap in Adversarial Attacks

Iterative attacks like Projected Gradient Descent (PGD) often produce "sparse" perturbations, changing only a few pixels significantly and resulting in a low $\ell_2$ distance (RMSE $\approx$ 7). In contrast, simpler or more aggressive attacks like FGSM push nearly every pixel toward the $\ell_\infty$ limit, resulting in much higher $\ell_2$ distances (RMSE $\approx$ 16). The success of many state-of-the-art transfer attacks is often not due to superior optimization, but simply because they produce perturbations with 70% to 130% larger $\ell_2$ distances.

The "Equalizer" Effect

To isolate the effect of the $\ell_2$ norm, the researchers introduced a modified loss function that allows for explicitly "dialing up" the RMSE while keeping the $\ell_\infty$ bound constant. The most striking finding is the "Equalizer" effect: when different attack algorithms are forced to produce the same $\ell_2$ distance, their transferability becomes almost identical. The specific optimization path matters far less than the total perturbation energy.

The Equalizer Effect: When the $\ell_2$ norm (RMSE) is fixed to a constant value (e.g., 15), the Attack Success Rate (ASR) curves for all tested attacks flatten out, demonstrating that $\ell_2$ is the dominant factor in transferability.

Visibility and Perceptibility Trade-offs

Larger $\ell_2$ distances make adversarial examples more perceivable to humans. By ignoring $\ell_2$, researchers may inadvertently favor "noisier" attacks that are easier to spot simply because they transfer better. This highlights the need for multi-norm measurements and the exploration of perceptual metrics like the Structural Similarity Index (SSIM) to ensure defenses are not just overfitting to low-energy attacks.

Visual comparison of adversarial examples (a-f) with varying $\ell_2$ norms. As the $\ell_2$ distance increases, the perturbations become significantly more visible to the human eye, despite all images sharing the same $\ell_\infty$ bound.

Conclusion

The $\ell_\infty$ norm is an incomplete metric for evaluating adversarial strength. To ensure truly unbiased and fair evaluations, researchers must measure and report both $\ell_\infty$ and $\ell_2$ distances. This shift is necessary to prevent the community from being misled by "tricky" enhancements that rely on increased perturbation energy rather than algorithmic innovation.

Summary updated

Abstract

Deep Neural Networks (DNNs) could be easily fooled by Adversarial Examples (AEs) with the imperceptible difference to original samples in human eyes. To keep the difference imperceptible, the existing attacking bound the adversarial perturbations by the $\ell_\infty$ norm, which is then served as the standard to align different attacks for a fair comparison. However, when investigating transferability, i.e., the capability of the AEs from attacking one surrogate DNN to cheat other black-box DNN, we find that only using the $\ell_\infty$ norm is not sufficient to measure the strength, according to our comprehensive experiments concerning 7 transfer-based attacks, 4 white-box surrogate models, and 9 black-box victim models. Specifically, we find that the $\ell_2$ norm greatly affects the in $\ell_\infty$ attacks. Since larger-perturbed AEs naturally bring about better transferability, we advocate that the of all attacks should be measured by both the widely used $\ell_\infty$ and also the $\ell_2$ norm. Despite the intuitiveness of our conclusion and advocacy, they are very necessary for the community, because common evaluations (bounding only the $\ell_\infty$ norm) allow tricky enhancements of the attack transferability by increasing the attack strength ($\ell_2$ norm) as shown by our simple counter-example method, and the good of several existing methods may be due to their large $\ell_2$ distances.
BibTeX
@misc{chen2021measuring,
  author = {Sizhe Chen and Qinghua Tao and Zhixing Ye and Xiaolin Huang},
  title = {{Measuring $\textbackslash\{\}ell\_\textbackslash\{\}infty$ Attacks by the $\textbackslash\{\}ell\_2$ Norm}},
  year = {2021},
  eprint = {2102.10343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.10343},
}
Figure