Probing SAM Perturbations Through Statistical Replay

Ting Han, Jens Kleesiek, Michael KamparXiv 2026 · arXiv

TL;DR

Researchers used a "statistical replay" technique to dissect Sharpness-Aware Minimization (SAM), finding that its generalization benefits cannot be explained by simple curvature regularization or the statistical distribution of its perturbations alone. While adding noise to SGD can match SAM's validation accuracy, it fails to replicate SAM's unique ability to maintain near-perfect training fit, suggesting the "magic" lies in the precise coupling between the perturbation and the model's trajectory.

What Problem Does This Paper Address?

In the world of deep learning, we don't just want models that memorize training data; we want models that generalize to new, unseen examples. Sharpness-Aware Minimization (SAM) has become a go-to optimizer for this. The intuition is simple: instead of just looking for the lowest point on the loss landscape, SAM looks for a "flat" neighborhood where the loss stays low even if you wiggle the weights slightly.

However, despite its success, we don't fully understand why it works so well. Is it just because it penalizes curvature? Is it the specific direction of the adversarial steps it takes? Or is it just a fancy way of adding noise? This paper attempts to "deconstruct" SAM to see which of its components are truly indispensable.

Methods

The authors introduce a diagnostic tool called Statistical Replay. The process works in two stages:

  1. Recording: They first train a "reference" model using standard SAM and record every adversarial perturbation ($\Delta_t$) the optimizer makes. They summarize these perturbations by calculating their element-wise means and standard deviations over local windows (e.g., every 10 minibatches).
  2. Replaying: They then train a completely new model using standard SGD but "replay" the recorded SAM statistics. Instead of calculating a new adversarial step, they sample a perturbation from the recorded means and variances and apply it.

They tested several variants:

  • Pure Replay: Uses only the recorded SAM statistics.
  • Mixed Replay: Combines SAM statistics with isotropic Gaussian noise.
  • Random Direction SAM: A crucial control where they keep SAM’s fixed radius but replace the adversarial direction with a completely random one.
  • Curvature Regularizers: They also tested explicit penalties on the Hessian trace and largest eigenvalue to see if direct curvature control could match SAM.

Datasets

The experiments were conducted using a ResNet-18 architecture on the CIFAR-10 dataset. Notably, the authors chose to train without data augmentation. This is a deliberate choice to isolate the effects of the optimizer itself, as data augmentation can often mask or overlap with the generalization benefits of SAM.

Key Insights and Ideas

  • Curvature isn't the whole story: Explicitly regularizing the Hessian trace or the largest eigenvalue (mathematical proxies for "sharpness") improved SGD but still performed significantly worse than SAM.
  • The "Adversarial" part is mandatory: In the "Random Direction SAM" experiment, using a random vector with the same radius as a SAM step resulted in performance no better than basic SGD. This proves that the direction of the perturbation—specifically its adversarial nature—is vital during the update.
  • SAM is an "Efficient" Noise: Mixed replay and pure isotropic noise can actually reach SAM-level validation accuracy if scaled correctly. However, there’s a catch: they significantly hurt training accuracy. SAM is unique because it achieves high validation performance while still allowing the model to fit the training data nearly perfectly (99.9%+ accuracy).
  • Alignment Matters: When the authors shuffled the timing of the replayed perturbations (temporal intervention), the benefits dropped. This suggests that the perturbations are not just generic noise; they are specifically "tuned" to the model's current position in the loss landscape.

Key Results

  • Standard SAM: Achieved ~80.5% validation accuracy with ~99.9% training accuracy.
  • Random Direction SAM: Dropped to ~76.5% validation accuracy, essentially matching standard SGD.
  • Pure Replay: Reached ~78.5% validation accuracy. It improved on SGD but couldn't catch SAM.
  • Mixed Replay: Could reach ~80.1% validation accuracy, but training accuracy fell to ~96.9%.
  • Isotropic Noise: Similarly reached ~80.1% validation accuracy, but training accuracy plummeted further to ~89.8% at higher noise scales.

Limitations and Future Work

The authors are transparent that these are preliminary results. The study is limited to one architecture (ResNet-18) and one dataset (CIFAR-10) without data augmentation.

The biggest takeaway is actually a "negative" result: many simple explanations for SAM (like "it's just noise" or "it's just trace regularization") don't hold up under scrutiny. The authors leave the door open for future research into directional correlations—how the two different gradient evaluations in a single SAM step interact over time—as the likely source of its unexplained power.

Summary updated

Abstract

Sharpness Aware Minimization (SAM) updates model parameters using gradients evaluated at adversarially perturbed weights, but it remains unclear which aspects of this procedure are sufficient to reproduce its generalization behavior. In our ResNet 18 experiments on CIFAR 10, simple curvature regularizers based on the Hessian trace and largest eigenvalue remain below SAM in validation accuracy, motivating a direct examination of its perturbations. We introduce statistical replay, a diagnostic that summarizes perturbations from reference SAM trajectories using elementwise means and standard deviations over local windows and constructs new perturbations during independent SGD training. Pure replay uses only these summaries, whereas mixed replay adds isotropic Gaussian noise. Pure replay does not reproduce SAM's validation behavior, while mixed replay and pure isotropic perturbations can approach its validation performance at selected scales without reproducing its training dynamics. An equal radius random direction within SAM performs similarly to SGD, and source, temporal, and layer interventions further suggest that replay behavior depends on perturbation construction. Together, these preliminary results weaken several simple explanations of SAM while leaving an important part of its mechanism unexplained.
BibTeX
@misc{han2026probing,
  author = {Ting Han and Jens Kleesiek and Michael Kamp},
  title = {{Probing SAM Perturbations Through Statistical Replay}},
  year = {2026},
  eprint = {2611.19480},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19480},
}
Figure