Fisher SAM: Information Geometry and Sharpness Aware Minimisation

Minyoung Kim, Da Li, Shell Xu Hu, Timothy M. Hospedales

Introduction

Modern deep learning models are highly overparameterized, allowing them to find numerous sets of weights (minima) that achieve zero training error. However, these minima are not created equal. "Sharp" minima—where the loss increases rapidly with slight parameter perturbations—tend to generalize poorly to unseen data. Conversely, "flat" minima are more robust to noise and typically yield better test performance.

Sharpness-Aware Minimization (SAM) was introduced to explicitly seek out these flat regions. It does so by minimizing the maximum loss within a small Euclidean neighborhood (a "ball") around the current parameters. While effective, SAM relies on a flawed assumption: that the neural network parameter space is Euclidean. In reality, it is a statistical manifold where a small change in one weight might barely affect the model's output, while a tiny change in another could drastically alter predictions. SAM’s rigid spherical neighborhood ignores this underlying geometry, potentially probing too far in sensitive directions or not far enough in insensitive ones.

The Fisher SAM (FSAM) Approach

To address this geometric mismatch, researchers introduced Fisher SAM (FSAM), which redefines the search neighborhood using the Fisher Information Matrix (FIM).

  1. Information Geometry: Instead of a fixed-radius L2 ball, FSAM utilizes the Kullback-Leibler (KL) divergence to define the neighborhood. This ensures that the "distance" moved in parameter space corresponds to a meaningful, proportional change in the model’s output distribution.
  2. Ellipsoidal Neighborhood: For small optimization steps, the KL divergence is approximated by a quadratic form using the FIM. This transforms SAM's spherical neighborhood into an ellipsoid that dynamically stretches or shrinks based on the model's sensitivity to specific parameters.
  3. Computational Efficiency: Calculating the full FIM is computationally prohibitive for deep networks. FSAM circumvents this by using a diagonalized minibatch approximation (specifically, the square of the batch gradient sum). This keeps the computational overhead nearly identical to the original SAM, requiring only two forward and backward passes per update.

Visualizing the Geometry: SAM vs. FSAM

The importance of accounting for parameter space geometry is best illustrated through a 2D toy landscape.

Comparison of SAM vs FSAM on a toy 2D landscape.

As seen above, SAM's circular neighborhood causes the optimizer to get "trapped" in a nearby sharp minimum. Because it searches equally in all directions, it cannot escape the steep walls of the local basin. In contrast, FSAM’s ellipsoidal neighborhood allows it to explore more aggressively along insensitive axes, successfully guiding the optimizer out of the sharp region and into a distant, flatter minimum.

Principled vs. Heuristic Adaptation

Previous attempts to improve SAM, such as Adaptive SAM (ASAM), relied on heuristics like scaling the neighborhood based on parameter magnitudes. While sometimes effective, this approach can fail when parameter magnitudes are small but sensitivity remains high.

Comparison of ASAM vs FSAM.

FSAM provides a more principled alternative. By evaluating the function's actual sensitivity via Fisher Information rather than just the parameter's scale, FSAM remains robust even when navigating near-zero parameter values where ASAM's heuristic breaks down.

Key Results and Performance

FSAM was evaluated across a wide range of standard benchmarks, demonstrating significant improvements over existing methods:

  • State-of-the-Art Accuracy: FSAM consistently outperformed SGD, SAM, and ASAM across tested architectures on CIFAR-10 and CIFAR-100. For instance, using a WideResNet-28-10, FSAM achieved 97.89% accuracy on CIFAR-10, compared to SAM's 97.56%.
  • Superior Robustness: The method proved significantly more resilient to adversarial parameter perturbations—tests where noise is intentionally added to weights in the worst possible direction. FSAM maintained higher accuracy as perturbation strength increased compared to its competitors.
  • Label Noise Resilience: In extreme scenarios with 80% symmetric label noise, FSAM maintained a test accuracy of 74.66%, vastly outperforming ASAM (70.56%) and SGD (66.32%).
  • Negligible Overhead: Despite the geometric calculations, the diagonal approximation keeps FSAM fast. On an RTX 2080 Ti, per-batch training time was 0.2334s, virtually indistinguishable from SAM's 0.2322s.

Limitations and Future Directions

While highly effective, FSAM's reliance on a diagonal FIM approximation means it ignores correlations between different parameters. Future work exploring block-diagonal or Kronecker-factored approximations could yield even better geometric representations, albeit at a higher computational cost. Additionally, integrating the FSAM loss with Natural Gradient Descent—which uses the inverse Fisher matrix to scale the update step itself—presents a promising avenue for further improving optimization in non-Euclidean spaces.

Summary updated

Abstract

Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by the maxi...
BibTeX
@inproceedings{kim2022fisher,
  author = {Minyoung Kim and Da Li and Shell Xu Hu and Timothy M. Hospedales},
  title = {{Fisher SAM: Information Geometry and Sharpness Aware Minimisation}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=52ia0URXYJf},
}
Figure