Assessing Local Generalization Capability in Deep Models

Huan Wang, Nitish Shirish Keskar, Caiming Xiong, Richard SocherAISTATS 2020 · OpenReview

Abstract

While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima, which can be described via the Hessian. We connect model generalization to the flatness of the loss landscape using the PAC-Bayes framework, deriving a new metric and optimization algorithm, PertOPT, which adaptively perturbs model weights during training to find smoother, more robust optima.

Introduction

A central mystery in deep learning is why massive models with millions of parameters generalize so well, even when they have the capacity to simply "memorize" the training set. Classical learning theory, such as VC-dimension, suggests these models should overfit significantly. However, empirical observations show the opposite.

Researchers have long hypothesized that "flat minima"—regions of the loss landscape where the error remains relatively constant despite small weight perturbations—are the key to generalization. Previous attempts to formalize this were often criticized because "sharpness" can be artificially manipulated by re-scaling weights (the re-parameterization problem). This research seeks to provide a theoretically sound connection between local smoothness and generalization that is robust to these issues.

Theoretical Framework: PAC-Bayes and the Hessian

The authors leverage the PAC-Bayes framework to bound the gap between training and test error. Their derivation proves that generalization capability is fundamentally tied to three factors:

  1. The Hessian: Second-order derivatives indicating the curvature of the loss landscape.
  2. The Hessian Lipschitz Constant: A third-order term describing how quickly the curvature changes.
  3. Parameter Scale: The magnitude of the learned weights.

Based on this proof, they propose pacGen, a metric to score a model's generalization capability. pacGen utilizes an Adam-like approximation to estimate the Hessian diagonal efficiently, making it practical for large-scale models.

The PertOPT Algorithm

To practically apply these theoretical insights, the authors introduce PertOPT (Perturbed Optimization). Unlike standard noise injection techniques, PertOPT adaptively determines the perturbation level for each parameter based on local geometry. It injects more noise into "flat" directions and less into "sharp" directions, effectively steering the optimizer toward regions that satisfy the derived PAC-Bayes bound. This geometry-aware approach acts as a sophisticated form of regularization.

Experimental Validation

The approach was validated across several benchmarks:

  • Toy 2D Example: A mixture of Gaussians used to visualize the loss landscape and prediction behavior.
  • MNIST: Demonstrated that the pacGen metric correctly tracks the generalization gap across varying batch sizes and learning rates.
  • CIFAR-10 & CIFAR-100: Evaluated PertOPT against standard optimizers (Adam/SGD) and Dropout using Wide-ResNet architectures.
  • Tiny ImageNet: Tested the scalability of the PertOPT approach on more complex data.

Discussion: Flatness and Robustness

The core insight of this work is that generalization is not just about the steepness of the valley (second-order), but also how smoothly that steepness changes (third-order). By forcing the model to remain performant under tailored weight perturbations, PertOPT avoids "sharp" minima that fit training data perfectly but fail on unseen samples.

Visualizing the relationship between loss landscape flatness and prediction accuracy. (a) Loss landscape showing sharp vs flat minima. (b) Sample distribution. (c) Predictions from a sharp minimum. (d) Predictions from a flat minimum.

Experimental results on CIFAR and Tiny ImageNet showed that PertOPT consistently outperformed both baseline optimizers and standard Dropout. For instance, on CIFAR-10, PertOPT achieved higher validation accuracy than any tested Dropout rate, suggesting that geometry-aware perturbations are superior to random neuron-dropping.

Conclusion and Future Work

While this work provides a rigorous link between flatness and generalization, some challenges remain. The current bound still "explodes" in extreme over-parameterized regimes where the number of parameters significantly exceeds the number of samples. Additionally, while efficient, estimating Hessian-related terms adds computational overhead compared to simple SGD.

Future research will focus on tightening these bounds for "infinite width" models and developing even more efficient methods to estimate the Hessian Lipschitz constant in real-time during training.

Summary updated

Abstract

While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima, which can be described via the Hessian. We connect model genera...
BibTeX
@inproceedings{wang2020assessing,
  author = {Huan Wang and Nitish Shirish Keskar and Caiming Xiong and Richard Socher},
  title = {{Assessing Local Generalization Capability in Deep Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=e6SpyUQlFW4},
}
Figure