Abstract
Recent sharpness-aware minimisation (SAM) is known to find flat minima which is beneficial for better generalisation with improved robustness. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood around the current iterate. However, it uses the Euclidean ball to define the neighborhood, which can be inaccurate since loss functions for neural networks are typically defined over probability distributions (e.g., class predictive probabilities), rendering the parameter space non-Euclidean. In this paper, we consider the information geometry of the model parameter space when defining the neighborhood, namely replacing SAM's Euclidean balls with ellipsoids induced by the Fisher information. Our approach, dubbed Fisher SAM, defines more accurate neighborhood structures that conform to the intrinsic metric of the underlying statistical manifold. For instance, SAM may probe the worst-case loss value at either a too nearby or inappropriately distant point due to the ignorance of the parameter space geometry, which is avoided by our Fisher SAM. We demonstrate improved performance of the proposed Fisher SAM on several benchmark datasets and tasks.
Introduction: The Search for Flatness
In deep learning, the goal is not merely to achieve low training error but to ensure the model generalizes well to unseen data. A growing body of research suggests that "flat" minima—regions in the loss landscape where the error remains relatively stable despite small parameter perturbations—generalize significantly better than "sharp" minima.
The popular Sharpness-Aware Minimization (SAM) algorithm explicitly searches for these flat regions by minimizing the maximum loss within a small neighborhood (a Euclidean ball) around the current parameters. However, the parameter space of a neural network is inherently non-Euclidean because networks output probability distributions. Standard SAM treats all directions identically, which can be inefficient or misleading when the loss landscape is highly curved or anisotropic.
Methodology: Fisher Information as a Metric
Fisher SAM (FSAM) improves upon SAM by incorporating Information Geometry to define a more principled neighborhood. Instead of a uniform Euclidean ball, FSAM utilizes an ellipsoid defined by the Fisher Information Matrix (FIM).
From Balls to Ellipsoids
The FIM acts as a metric tensor that describes how sensitive the model's output distribution is to changes in its parameters. By defining the neighborhood via the Kullback-Leibler (KL) divergence between the current model and a perturbed version, FSAM effectively "stretches" the search area in directions where parameters have little impact on the output and "shrinks" it where parameters are highly sensitive.
Computational Efficiency
Calculating the full FIM is computationally prohibitive for modern deep networks. To maintain the speed of standard SAM, FSAM employs a diagonalized minibatch approximation. By using the square of the batch gradient sum, the authors achieve the benefits of information geometry with negligible overhead—training a batch on an RTX 2080 Ti takes approximately 0.2334s for FSAM compared to 0.2322s for SAM.
Navigating the Curved Landscape
A key advantage of FSAM is its ability to "probe" the landscape more effectively. In toy 2D experiments, standard SAM often becomes trapped in sharp minima because its circular neighborhood fails to capture the path toward flatter regions. In contrast, FSAM’s ellipsoidal neighborhood adapts to the local curvature, allowing the optimizer to escape sharp valleys and find more robust, flatter minima.
Empirical Evaluation and Robustness
The authors evaluated FSAM across several computer vision benchmarks, including CIFAR-10/100, ImageNet (fine-tuning DeiT-B), and transfer learning tasks like Stanford Cars.
- Superior Generalization: FSAM consistently outperformed SGD, SAM, and Adaptive SAM (ASAM). For instance, on CIFAR-100 with a PyramidNet-272 architecture, FSAM achieved 86.93% accuracy, surpassing SAM's 85.19%.
- Resilience to Label Noise: In experiments with 80% label noise on CIFAR-10, FSAM maintained 74.66% accuracy, significantly higher than SGD (66.32%) and ASAM (70.56%).
- Parameter Robustness: When final model weights were subjected to adversarial perturbations, FSAM models degraded much more slowly than those trained with SAM, confirming that the minima found were indeed flatter and more stable.
Conclusion
Fisher SAM provides a theoretically grounded and computationally efficient extension to sharpness-aware optimization. By recognizing that parameter magnitude is a poor proxy for functional importance, FSAM leverages the Fisher Information Matrix to navigate the intrinsic geometry of neural networks. While the current implementation relies on a diagonal approximation of the FIM, the results demonstrate that even this simplified geometric awareness leads to significantly better generalization and robustness across a variety of deep learning tasks.
Summary updated