Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss

Yunrui Yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun Zhu

Abstract

Gradient-based adversarial attacks using the Cross-Entropy (CE) loss often suffer from overestimation due to relative errors in gradient computation induced by floating-point arithmetic. This paper provides a rigorous theoretical analysis of these errors, conducting the first comprehensive study of floating-point computation errors in gradient-based attacks across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. We establish theoretical foundations characterizing the behavior of relative numerical errors under different attack conditions, revealing previously unknown patterns in gradient computation instability, and identify floating-point underflow and rounding as key contributors. Building on this insight, we propose the Theoretical MIFPE (T-MIFPE) loss function, which incorporates an optimal scaling factor $T = t^*$ to minimize the impact of floating-point errors, thereby enhancing the accuracy of gradient computation in adversarial attacks. Extensive experiments on the MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate that T-MIFPE outperforms existing loss functions, including CE, C&W, DLR, and MIFPE, in terms of attack potency and robustness evaluation accuracy.

Introduction: The Gradient Masking Problem

In adversarial machine learning, evaluating a model's robustness involves attempting to find small perturbations that cause misclassification. The Cross-Entropy (CE) loss is the standard tool for guiding these attacks. However, CE loss often leads to "robustness overestimation." This occurs not because the model is inherently robust, but because the gradients used to find adversarial examples become numerically unstable.

Floating-point arithmetic on modern hardware can lead to underflow (values becoming zero) or significant rounding errors when dealing with the very small or very large values typical of the CE loss's tail. This creates a form of "gradient masking," where the attacker receives incorrect directional information, leading to failed attacks on vulnerable models.

Theoretical Analysis of Numerical Errors

The authors provide the first rigorous analysis of these numerical errors by categorizing the attack process into four distinct phases. This systematic breakdown allows for a precise characterization of how floating-point errors evolve as an attack progresses toward success or failure.

Analysis of relative numerical errors across four attack scenarios.

The analysis reveals that the relative error is not constant but depends heavily on the model's current prediction state (the logits). By understanding these patterns, the researchers identified that the instability is primarily driven by the interaction between the loss function's mathematical form and the limitations of 16-bit, 32-bit, and 64-bit floating-point representations.

T-MIFPE and the Optimal Scaling Factor

Building on this theoretical foundation, the authors propose the Theoretical MIFPE (T-MIFPE) loss function. While previous work suggested scaling logits to mitigate errors, T-MIFPE introduces a dynamic, mathematically derived optimal scaling factor, $t^*$.

Optimal scaling factor $t^*$ in an unsuccessful untargeted attack.

Unlike static scaling methods, $t^*$ is recalculated at each step of the attack. This ensures that the logits remain in a "Goldilocks zone" where gradients are neither too small (avoiding underflow) nor too large (avoiding rounding saturation), keeping the relative numerical error at its absolute minimum.

Experimental Results and Efficiency

The researchers evaluated T-MIFPE on benchmark datasets including MNIST, CIFAR-10, CIFAR-100, and ImageNet against various defended models. Key findings include:

  • Superior Potency: T-MIFPE consistently found successful adversarial examples that standard CE and other specialized losses like C&W or DLR missed.
  • High Efficiency: T-MIFPE achieves accuracy comparable to the "gold standard" AutoAttack (which uses ~5,000 iterations) in only 100 iterations. For example, on a defended CIFAR-10 model, T-MIFPE reached a robust accuracy of 65.95%, nearly matching the 65.87% benchmark.
  • Universal Improvement: Unlike some alternatives that only work well on specific architectures, T-MIFPE showed universal improvement across all tested models.

Conclusion

This work highlights that floating-point arithmetic is a critical bottleneck in AI security evaluation. By providing a theoretical framework for numerical errors and deriving the optimal T-MIFPE loss, the authors have created a tool that makes adversarial robustness testing both more accurate and significantly faster. Future work will look to extend these large-scale benchmarks to targeted attack scenarios and explore the implications for robust training.

Summary updated

Abstract

Gradient-based adversarial attacks using the Cross-Entropy (CE) loss often suffer from overestimation due to relative errors in gradient computation induced by floating-point arithmetic. This paper provides a rigorous theoretical analysis of these errors, conducting the first comprehensive study of floating-point computation errors in gradient-based attacks across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. We establish theoretical foundations characterizing the behavior of relative numerical errors under different attack conditions, revealing previously unknown patterns in gradient computation instability, and identify floating-point underflow and rounding as key contributors. Building on this insight, we propose the Theoretical MIFPE (T-MIFPE) loss function, which incorporates an optimal scaling factor $T = t^*$ to minimize the impact of floating-point errors, thereby enhancing the accuracy of gradient computation in adversarial attacks. Extensive experiments on the MNIST, CIFAR-10, and CIFAR-100 datasets demonstrate that T-MIFPE outperforms existing loss functions, including CE, C\&W, DLR, and MIFPE, in terms of attack potency and robustness evaluation accuracy.
BibTeX
@misc{yu2025theoretical,
  author = {Yunrui Yu and Hang Su and Cheng-zhong Xu and Zhizhong Su and Jun Zhu},
  title = {{Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss}},
  year = {2025},
  eprint = {2507.22428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.22428},
}
Figure