Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss

yunrui yu, Hang Su, Cheng-zhong Xu, Zhizhong Su, Jun ZhuSubmission to ICLR 2026 · Rejected

TL;DR

Standard Cross-Entropy (CE) loss causes gradient-based adversarial attacks to overestimate model robustness due to numerical underflow and truncation errors in floating-point arithmetic. This paper provides the first rigorous theoretical error analysis across four attack phases, derives the optimal logit-scaling factor ($t^*$), and theoretically justifies why prior empirical fixes (like MIFPE’s $T=1$) worked so well. By replacing the heuristic scaling with the dynamically derived $t^*$, the authors propose T-MIFPE, matching or beating expensive attack ensembles like AutoAttack with significantly fewer iterations.

What Problem Does This Paper Address?

Evaluating how vulnerable a deep neural network is to adversarial perturbations usually relies on white-box attacks like Projected Gradient Descent (PGD). However, when paired with standard cross-entropy loss, gradient descent routinely overestimates model robustness—falsely declaring a defense secure when it is not.

Prior work discovered that this failure is partly a numerical artifact: in standard 32-bit floating-point arithmetic, computing gradients through the softmax and CE loss incurs severe relative errors due to vanishing gradients, rounding, and underflow. An empirical patch called MIFPE previously addressed this by scaling logits by a factor of $c = T / \Delta_{\text{detach}}$ (where $\Delta_{\text{detach}}$ is the logit margin between the top two classes and $T=1$). While MIFPE worked remarkably well, nobody knew why $T=1$ was the right choice or how numerical error behaves dynamically across different attack stages.

Methods

The authors decompose gradient attacks into four distinct mathematical scenarios depending on the goal and current classification state:

  1. Unsuccessful untargeted attacks (logits predict the correct class; attack is attempting to misclassify).
  2. Successful untargeted attacks (misclassification achieved; attack is pushing further).
  3. Unsuccessful targeted attacks (target logit is not yet the maximum).
  4. Successful targeted attacks (target logit is the maximum).

For each scenario, they:

  • Formulate the Relative Gradient Error: They derive the gradient components that drive the attack objective and express the supremum of relative floating-point error as $\delta(t)^{\text{sup}} = \epsilon_{\text{max}} / r$, where $r$ is the exact gradient magnitude and $\epsilon_{\text{max}}$ is machine precision ($2^{-23}$ for float32).
  • Maximize the Denominator: Minimizing the relative error bound corresponds to maximizing a scalar function $g(t)$. For instance, in unsuccessful untargeted attacks, $g(t)_{u\_u} = c(1 - p_{\pi_1}^c)$.
  • Derive Bounds and Find $t^*$: The authors bound $g(t)$ between lower and upper curves across class counts $K \in \{10, 100, 1000\}$ and compute the optimal scaling factor $t^* = \arg\max g(t)$ via an ultra-fast GPU grid search (taking under 0.05 seconds for 10,000 samples) or closed-form underflow thresholds.
  • Introduce T-MIFPE: They formulate the Theoretical MIFPE (T-MIFPE) loss, substituting static scaling with the scenario-adaptive, iteration-dependent $t^*$.

Datasets

The authors evaluate attacks on standard adversarial robustness benchmarks:

  • CIFAR-10 ($\ell_\infty, \epsilon = 8/255$)
  • CIFAR-100 ($\ell_\infty, \epsilon = 8/255$)
  • ImageNet ($\ell_\infty, \epsilon = 4/255$)

Evaluations are performed against widely recognized robust models from the RobustBench leaderboard (e.g., Gowal et al., Rebuffi et al., Wu et al., Sitawarin et al., Wong et al., and Salman et al.).

Key Insights and Ideas

  • The Mystery of $T=1$ Solved: For unsuccessful untargeted attacks, the lower bound of $g(t)$ peaks at $t \approx 1.278$. Empirical distributions of optimal $t^*$ values cluster tightly around this point. This provides mathematical justification for why MIFPE’s empirical choice of $T=1$ was effective—it sat right next to the theoretical optimum.
  • Error Dynamics Shift Post-Misclassification: In successful phases or targeted attacks, error behavior changes drastically. Rather than having an interior peak, the objective functions often increase monotonically with $t$ until bounded by numerical underflow thresholds ($\lambda$).
  • Gradients Suffer Even When Defenses Do Not Intend Gradient Masking: Robustness overestimation is frequently blamed on model architectures deliberately masking gradients, but this work proves that basic IEEE-754 precision limitations mask gradients unintentionally unless logits are normalized.

Key Results

  • Untargeted PGD-100 Attacks: T-MIFPE consistently improves robustness evaluation accuracy over standard CE and achieves modest gains over MIFPE:
  • On ImageNet (Wong et al., 2020), T-MIFPE reduces evaluated robust accuracy to 25.38% vs. MIFPE's 25.71% and CE's 27.12%.
  • On CIFAR-100 (Sitawarin et al., 2020), robust accuracy drops to 24.96% vs. MIFPE's 25.24% and CE's 26.18%.
  • Matching AutoAttack at Lower Cost: In multi-targeted attacks (900 total iterations), T-MIFPE matches or outperforms the standard benchmark AutoAttack (which requires 4,900 iterations across an ensemble):
  • On ImageNet (Wong et al., 2020): T-MIFPE achieves 25.02% robust accuracy compared to AutoAttack’s 25.24% (a 0.22% improvement in attack efficacy).
  • On CIFAR-100 (Rice et al., 2020): T-MIFPE reaches 18.92% vs. AutoAttack's 18.95%.

Limitations and Future Work

  • Marginal Empirical Gains over MIFPE: Because MIFPE's heuristic choice of $T=1$ is already extremely close to the theoretical optimum ($t^* \approx 1.278$), the empirical accuracy drop provided by T-MIFPE over MIFPE is modest (typically between 0.01% and 0.34%). The authors openly acknowledge that their contribution is primarily establishing a rigorous theoretical foundation rather than delivering an order-of-magnitude empirical jump.
  • Scope of Loss Functions: The analysis is focused strictly on Cross-Entropy loss variants and bounded $\ell_\infty$ attacks; extending the numerical precision analysis to margin-based losses (like C&W or DLR) or other perturbation norms ($\ell_1$, $\ell_0$) remains future work.

Summary updated

Abstract

Gradient-based adversarial attacks using the Cross-Entropy (CE) loss often overestimate robustness due to relative errors in gradient computation induced by floating-point arithmetic. Empirical methods like MIFPE mitigate this by scaling logits with a factor $ c = T / \Delta_{\text{detach}} $ where $ T = 1 $, significantly improving evaluation accuracy. However, a theoretical understanding of these errors remains limited. To bridge this gap, we pioneer the first rigorous theoretical analysis of floating-point errors in CE-based gradient attacks, systematically dissecting relative errors across four distinct scenarios: (i) unsuccessful untargeted attacks, (ii) successful untargeted attacks, (iii) unsuccessful targeted attacks, and (iv) successful targeted attacks. This foundational study uncovers novel patterns in numerical instability and derives the optimal scaling factor $T = t^\* $ that minimizes error impact in each scenario. Notably, our analysis reveals that $ t^\* $ closely approximates 1 in unsuccessful untargeted attacks, providing a theoretical justification for MIFPE's empirical choice and addressing prior optimality gaps. To validate the correctness of our theoretical derivations, we refine MIFPE by incorporating $ T = t^\* $ into the Theoretical MIFPE (T-MIFPE) loss function, which further reduces floating-point-induced errors. Comprehensive experiments validate our theory.
BibTeX
@misc{yu2025theoretical,
  author = {yunrui yu and Hang Su and Cheng-zhong Xu and Zhizhong Su and Jun Zhu},
  title = {{Theoretical Analysis of Relative Errors in Gradient Computations for Adversarial Attacks with CE Loss}},
  year = {2025},
  eprint = {2507.22428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.22428},
}
Figure