Fisher SAM: Information Geometry and Sharpness Aware Minimisation

Minyoung Kim, Da Li, Shell Xu Hu, Timothy M. Hospedales

Abstract

Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood around the current iterate. However, it uses the Euclidean ball to define the neighborhood, which can be inaccurate since loss functions for neural networks are typically defined over probability distributions (e.g., class predictive probabilities), rendering the parameter space non-Euclidean. In this paper, we consider the information geometry of the model parameter space when defining the neighborhood, namely replacing SAM's Euclidean balls with ellipsoids induced by the Fisher information. Our approach, dubbed Fisher SAM, defines more accurate neighborhood structures that conform to the intrinsic metric of the underlying statistical manifold. For instance, SAM may probe the worst-case loss value at either a too nearby or inappropriately distant point due to the ignorance of the parameter space geometry, which is avoided by our Fisher SAM. We demonstrate improved performance of the proposed Fisher SAM on several benchmark datasets and tasks.

Introduction: The Search for Flatness

In deep learning, the goal is not merely to achieve low training error but to ensure the model generalizes well to unseen data. A growing body of research suggests that "flat" minima—regions in the loss landscape where the error remains relatively stable despite small parameter perturbations—generalize significantly better than "sharp" minima.

Loss landscape with sharp (left) and flat (right) minima.

The popular Sharpness-Aware Minimization (SAM) algorithm explicitly searches for these flat regions by minimizing the maximum loss within a small neighborhood (a Euclidean ball) around the current parameters. However, the parameter space of a neural network is inherently non-Euclidean because networks output probability distributions. Standard SAM treats all directions identically, which can be inefficient or misleading when the loss landscape is highly curved or anisotropic.

Methodology: Fisher Information as a Metric

Fisher SAM (FSAM) improves upon SAM by incorporating Information Geometry to define a more principled neighborhood. Instead of a uniform Euclidean ball, FSAM utilizes an ellipsoid defined by the Fisher Information Matrix (FIM).

From Balls to Ellipsoids

The FIM acts as a metric tensor that describes how sensitive the model's output distribution is to changes in its parameters. By defining the neighborhood via the Kullback-Leibler (KL) divergence between the current model and a perturbed version, FSAM effectively "stretches" the search area in directions where parameters have little impact on the output and "shrinks" it where parameters are highly sensitive.

Computational Efficiency

Calculating the full FIM is computationally prohibitive for modern deep networks. To maintain the speed of standard SAM, FSAM employs a diagonalized minibatch approximation. By using the square of the batch gradient sum, the authors achieve the benefits of information geometry with negligible overhead—training a batch on an RTX 2080 Ti takes approximately 0.2334s for FSAM compared to 0.2322s for SAM.

A key advantage of FSAM is its ability to "probe" the landscape more effectively. In toy 2D experiments, standard SAM often becomes trapped in sharp minima because its circular neighborhood fails to capture the path toward flatter regions. In contrast, FSAM’s ellipsoidal neighborhood adapts to the local curvature, allowing the optimizer to escape sharp valleys and find more robust, flatter minima.

Comparison of optimization trajectories: SAM (left) gets trapped in a sharp minimum, while Fisher SAM (right) navigates to the flatter, more desirable minimum.

Empirical Evaluation and Robustness

The authors evaluated FSAM across several computer vision benchmarks, including CIFAR-10/100, ImageNet (fine-tuning DeiT-B), and transfer learning tasks like Stanford Cars.

  • Superior Generalization: FSAM consistently outperformed SGD, SAM, and Adaptive SAM (ASAM). For instance, on CIFAR-100 with a PyramidNet-272 architecture, FSAM achieved 86.93% accuracy, surpassing SAM's 85.19%.
  • Resilience to Label Noise: In experiments with 80% label noise on CIFAR-10, FSAM maintained 74.66% accuracy, significantly higher than SGD (66.32%) and ASAM (70.56%).
  • Parameter Robustness: When final model weights were subjected to adversarial perturbations, FSAM models degraded much more slowly than those trained with SAM, confirming that the minima found were indeed flatter and more stable.

Conclusion

Fisher SAM provides a theoretically grounded and computationally efficient extension to sharpness-aware optimization. By recognizing that parameter magnitude is a poor proxy for functional importance, FSAM leverages the Fisher Information Matrix to navigate the intrinsic geometry of neural networks. While the current implementation relies on a diagonal approximation of the FIM, the results demonstrate that even this simplified geometric awareness leads to significantly better generalization and robustness across a variety of deep learning tasks.

Summary updated

Abstract

Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood around the current iterate. However, it uses the Euclidean ball to define the neighborhood, which can be inaccurate since loss functions for neural networks are typically defined over probability distributions (e.g., class predictive probabilities), rendering the parameter space non Euclidean. In this paper we consider the information geometry of the model parameter space when defining the neighborhood, namely replacing SAM's Euclidean balls with ellipsoids induced by the Fisher information. Our approach, dubbed Fisher SAM, defines more accurate neighborhood structures that conform to the intrinsic metric of the underlying statistical manifold. For instance, SAM may probe the worst-case loss value at either a too nearby or inappropriately distant point due to the ignorance of the parameter space geometry, which is avoided by our Fisher SAM. Another recent Adaptive SAM approach stretches/shrinks the Euclidean ball in accordance with the scale of the parameter magnitudes. This might be dangerous, potentially destroying the neighborhood structure. We demonstrate improved performance of the proposed Fisher SAM on several benchmark datasets/tasks.
BibTeX
@inproceedings{kim2022fisher,
  author = {Minyoung Kim and Da Li and Shell Xu Hu and Timothy M. Hospedales},
  title = {{Fisher SAM: Information Geometry and Sharpness Aware Minimisation}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=52ia0URXYJf},
}
Figure