CR-SAM: Curvature Regularized Sharpness-Aware Minimization

Tao Wu, Tie Luo, Donald C. Wunsch IiAAAI 2024 · DOI

TL;DR

This paper introduces CR-SAM, a novel method to improve deep learning model generalization by better capturing the curvature of the loss landscape during training. CR-SAM addresses the limitations of existing Sharpness-Aware Minimization (SAM) techniques, which struggle with the increasing non-linearity of loss landscapes in later training stages, leading to more robust and accurate models.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful, but their real-world utility hinges on their ability to generalize to unseen data. A key factor influencing generalization is the "flatness" of the loss landscape. Flatter minima are generally associated with better generalization. Sharpness-Aware Minimization (SAM) is a popular technique that aims to find these flatter minima by minimizing the maximum loss within a small neighborhood around the current model parameters.

However, SAM relies on a one-step gradient ascent to approximate this "worst-case" loss. As training progresses, the loss landscape becomes more complex and non-linear, making this one-step approximation less accurate. This degradation in accuracy means SAM's effectiveness diminishes over time. Furthermore, traditional metrics for measuring curvature, like the Hessian trace or its largest eigenvalue, also become unreliable in later training stages, failing to accurately reflect the increasing non-linearity. This leaves a gap in effectively guiding DNNs towards truly flat and generalizable minima.

Methods

The core of CR-SAM lies in a new metric for characterizing loss landscape curvature: the normalized Hessian trace. This metric, defined as the trace of the Hessian divided by the squared norm of the gradient, is shown to consistently reflect the true curvature of the loss landscape, even as training progresses and traditional metrics falter.

To integrate this into a training framework, CR-SAM proposes a Curvature Regularized SAM (CR-SAM) objective. This objective penalizes both the normalized Hessian trace and the gradient norm, with a specific weighting to ensure it leads to flatter minima without negatively impacting generalization.

For computational efficiency, especially with large DNNs, CR-SAM employs a finite difference (FD) approach to approximate the normalized Hessian trace. This method allows for parallel computation of necessary function evaluations, making it scalable. The paper also provides theoretical backing for CR-SAM's effectiveness using PAC-Bayes bounds, demonstrating its ability to reduce generalization error.

Datasets

The paper evaluates CR-SAM on several standard image classification benchmarks:

  • CIFAR-10 and CIFAR-100: Smaller-scale datasets commonly used for initial model evaluation.
  • ImageNet-1k, ImageNet-C, and ImageNet-R: Larger-scale datasets, including out-of-distribution variants (ImageNet-C and ImageNet-R) to test robustness.

Key Insights and Ideas

  • One-step SAM approximation degrades: The paper empirically demonstrates that the one-step gradient ascent used in SAM becomes significantly less accurate in later training stages, directly impacting its ability to find flat minima.
  • Traditional curvature metrics fail: Hessian trace and top eigenvalue of the Hessian are shown to be unreliable indicators of curvature during SAM training, especially on test sets, due to their sensitivity to gradient scaling.
  • Normalized Hessian trace is key: The proposed normalized Hessian trace metric consistently tracks the increasing curvature of the loss landscape, providing a more accurate signal for regularization.
  • CR-SAM balances curvature and gradient: The regularization strategy in CR-SAM carefully balances penalizing curvature and gradient norm to achieve flatter minima without adverse effects.
  • Finite differences for scalability: The use of finite differences for computing the Hessian trace is crucial for making the method practical for large-scale DNNs.

Key Results

  • Consistent performance gains: CR-SAM consistently outperforms both standard SGD and vanilla SAM across various architectures (ResNet, ViT) and datasets (CIFAR-10/100, ImageNet-1k/-C/-R).
  • Significant accuracy improvements: For example, CR-SAM achieves up to a 1.11% accuracy improvement on CIFAR-100 with ResNet-18 and a 1.77% improvement for ViT-B/32 on ImageNet compared to SAM.
  • Flatter loss landscapes: Visualizations show that CR-SAM leads to demonstrably flatter loss landscapes compared to SGD and SAM.
  • Faster convergence: CR-SAM exhibits faster convergence rates than vanilla SAM, attributed to its ability to mitigate excessive curvature.
  • Reduced gradient norms and Hessian traces: Empirical analysis shows CR-SAM significantly reduces both gradient norms and Hessian traces, leading to a smaller normalized Hessian trace.

Limitations and Future Work

  • Computational cost: While the finite difference method improves efficiency, computing the Hessian trace still adds computational overhead compared to standard SGD. The paper mentions this is addressed by parallelization and efficient approximation.
  • Hyperparameter tuning: The CR-SAM objective introduces new hyperparameters (α and β) that require tuning, though the paper suggests a strategy for this.
  • Future work: The authors suggest that their approach can be seamlessly integrated with other SAM variants, implying potential for further improvements by combining CR-SAM with existing efficiency-focused SAM methods. They also highlight the potential for CR-SAM to be applied to other domains beyond image classification.

Summary updated

BibTeX
@inproceedings{wu2024cr,
  author = {Tao Wu and Tie Luo and Donald C. Wunsch II},
  title = {{CR-SAM: Curvature Regularized Sharpness-Aware Minimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28431},
}
Figure