Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless lp Norm Solution for Fast Adversarial Training

Fares B. Mehouachi, Saif Eddin JabariarXiv 2025 · DOI · arXiv

Abstract

Fast adversarial training using the Fast Gradient Sign Method (FGSM) is often hindered by Catastrophic Overfitting (CO), where models become vulnerable to multi-step attacks despite appearing robust to single-step ones. We propose a "noiseless" solution that adaptively tunes the $l^p$ norm of perturbations. By utilizing the "Participation Ratio" from quantum mechanics and "Entropy Gap" from information theory, our adaptive $l^p$-FGSM algorithm prevents model collapse without the need for computationally expensive regularization or random noise. Our method demonstrates superior robustness on ImageNet-1k while maintaining the efficiency of single-step training.

Introduction

Adversarial training is the primary defense against malicious input perturbations in Deep Neural Networks (DNNs). While iterative methods like Projected Gradient Descent (PGD) are effective, their computational cost is high. FGSM offers a faster alternative but is prone to Catastrophic Overfitting (CO). This paper investigates CO as a geometric mismatch and proposes an adaptive $l^p$ norm solution to bridge the gap between efficiency and robustness.

Understanding Catastrophic Overfitting

CO is a "false sense of security" where a model learns to defeat a specific training attack rather than becoming truly robust. This phenomenon is often accompanied by a sharp increase in gradient concentration, where the model's sensitivity becomes localized in just a few input dimensions.

Catastrophic Overfitting (CO) in FGSM training. Top: Test accuracy against PGD-50 drops sharply (CO) while FGSM accuracy rises. Bottom: CO is correlated with a spike in gradient norms (||∇θℓ||₂).

As illustrated above, the divergence between FGSM and PGD accuracy marks the onset of CO. The corresponding spike in gradient norms suggests that monitoring gradient statistics can provide an early warning system for model collapse.

Methodology: Adaptive $l^p$-FGSM

The proposed framework adaptively selects the perturbation norm $p$ between 2 (spherical) and $\infty$ (cubic) to maintain a smooth loss landscape. This approach treats adversarial attack generation as a fixed-point problem.

Metrics for Gradient Concentration

To detect the onset of CO, the authors adapt two cross-disciplinary metrics:

  • Participation Ratio (PR): Adapted from quantum mechanics, PR measures how many dimensions of a gradient vector meaningfully contribute to its structure. A low PR indicates high concentration.
  • Entropy Gap ($\Delta H$): An information-theoretic measure of how "spread out" gradient information is across the input space.

Adaptive Norm Selection

The algorithm dynamically calculates an optimal $p^*$ for each training step. If gradients begin to concentrate (signaling the onset of CO), the system lowers $p$ to maintain alignment with the natural geometry of the loss landscape.

Accuracy landscapes across different lp norms (p) and training epochs. CO is most prevalent at higher p values (approaching l∞), while lower p values offer more stability.

The 3D visualizations in the figure above show that CO is a geometric mismatch most prevalent under the $l^\infty$ norm. By using $l^p$ as a smooth proxy, the adaptive method finds a "sweet spot" that provides the robustness of $l^\infty$ with the stability of $l^2$.

Key Results and Insights

  • Superior Robustness: On ImageNet-1k, adaptive $l^p$-FGSM outperformed the popular RS-FGSM by approximately 3.3% in adversarial accuracy at higher perturbation magnitudes ($\epsilon = 6/255$).
  • Deterministic Stability: Unlike noise-based methods, this approach is "noiseless" and deterministic, solving CO by changing the geometry of the constraint ball rather than adding "fuzziness" to the data.
  • Efficiency: The computational overhead is minimal, as $l^p$ calculations are element-wise and based on existing gradient statistics, keeping the training process fast.

Conclusion

Adaptive $l^p$-FGSM provides a robust, efficient, and noiseless solution to the problem of Catastrophic Overfitting. By leveraging insights from quantum mechanics and information theory, this method allows for fast adversarial training that scales to large datasets like ImageNet without sacrificing model integrity or clean accuracy.

Summary updated

Abstract

Adversarial training is a cornerstone of robust deep learning, but fast methods like the Fast Gradient Sign Method (FGSM) often suffer from Catastrophic Overfitting (CO), where models become robust to single-step attacks but fail against multi-step variants. While existing solutions rely on noise injection, regularization, or gradient clipping, we propose a novel solution that purely controls the $l^p$ training norm to mitigate CO. Our study is motivated by the empirical observation that CO is more prevalent under the $l^{\infty}$ norm than the $l^2$ norm. Leveraging this insight, we develop a framework for generalized $l^p$ attack as a fixed point problem and craft $l^p$-FGSM attacks to understand the transition mechanics from $l^2$ to $l^{\infty}$. This leads to our core insight: CO emerges when highly concentrated gradients where information localizes in few dimensions interact with aggressive norm constraints. By quantifying gradient concentration through Participation Ratio and entropy measures, we develop an adaptive $l^p$-FGSM that automatically tunes the training norm based on gradient information. Extensive experiments demonstrate that this approach achieves strong robustness without requiring additional regularization or noise injection, providing a novel and theoretically-principled pathway to mitigate the CO problem.
BibTeX
@misc{mehouachi2025catastrophic,
  author = {Fares B. Mehouachi and Saif Eddin Jabari},
  title = {{Catastrophic Overfitting, Entropy Gap and Participation Ratio: A Noiseless lp Norm Solution for Fast Adversarial Training}},
  year = {2025},
  eprint = {2505.02360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.02360},
}
Figure