SmoothOut: Smoothing Out Sharp Minima for Generalization in Large-Batch Deep Learning

Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Yiran Chen, Hai LiarXiv 2018 · OpenReview

Abstract

In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It becomes an open question whether escaping sharp minima can improve the generalization. To answer this question, we propose SmoothOut framework to smooth out sharp minima in DNNs and thereby improve generalization. In a nutshell, SmoothOut perturbs multiple copies of the DNN by noise injection and averages these copies. Injecting noises to SGD is widely used in the literature, but SmoothOut differs in lots of ways: (1) a de-noising process is applied before parameter updating; (2) noise strength is adapted to filter norm; (3) an alternative interpretation on the advantage of noise injection, from the perspective of sharpness and generalization; (4) usage of uniform noise instead of Gaussian noise. We prove that SmoothOut can eliminate sharp minima. Training multiple DNN copies is inefficient, we further propose an unbiased stochastic SmoothOut which only introduces the overhead of noise injecting and de-noising per batch. An adaptive variant of SmoothOut, AdaSmoothOut, is also proposed to improve generalization. In a variety of experiments, SmoothOut and AdaSmoothOut consistently improve generalization in both small-batch and large-batch training on the top of state-of-the-art solutions.

Introduction

Large-batch training in deep learning is highly desirable for its computational efficiency on modern hardware. However, it is plagued by a persistent "generalization gap"—the phenomenon where models trained with large batches perform significantly worse on unseen data compared to those trained with small batches.

The prevailing explanation for this gap is the nature of the loss landscape. Large-batch training tends to converge to sharp minima, which are narrow, steep valleys. These regions are sensitive to small shifts in data distribution, leading to poor generalization. In contrast, small-batch training is inherently noisier, which encourages the optimizer to find flat minima that are more robust. SmoothOut is a novel framework designed to explicitly smooth the loss landscape, guiding the optimizer toward these desirable flat regions.

Methodology: Smoothing the Loss Landscape

The core principle of SmoothOut is to optimize an auxiliary loss function that represents a smoothed version of the original training loss.

The SmoothOut Framework

Conceptually, SmoothOut perturbs multiple copies of a model with noise and averages their outputs. This averaging process acts as a low-pass filter, effectively "erasing" high-frequency sharp minima while preserving the broad, flat regions that generalize better.

Stochastic Implementation and De-noising

To avoid the computational cost of training multiple model copies, the authors propose Stochastic SmoothOut. In each iteration, noise $\theta_t$ is injected into the parameters $w_t$. After calculating the gradient on the perturbed weights, a critical de-noising step is performed: the noise is subtracted ($w_t - \theta_t$) before the parameter update is applied. This ensures the optimizer is truly navigating the smoothed landscape rather than just adding random jitter.

The SmoothOut framework: (a) original loss landscape with flat and sharp minima, (b) the smoothing effect of averaging, (c) the conceptual parallel version, and (d) the efficient stochastic version.

Adaptive Noise and Uniform Distribution

SmoothOut employs uniform noise rather than the traditional Gaussian noise. Theoretical proofs in the paper demonstrate that uniform noise provides a more effective "hard boundary" for lifting the floor of sharp minima. Furthermore, AdaSmoothOut scales the noise strength based on the $L_2$ norm of the filters, ensuring that noise levels are appropriate for the scale of weights in different layers.

Experimental Evaluation

The framework was tested on several benchmarks, including MNIST, CIFAR-10, CIFAR-100, and ImageNet, using architectures such as ResNet and AlexNet.

Key Results

  • Closing the Generalization Gap: On CIFAR-10, SmoothOut improved the accuracy of a large-batch (5000) model by 3.04%, significantly narrowing the gap with small-batch performance.
  • Sharpness Reduction: Empirical measurements showed that SmoothOut reduces the sharpness of final minima by several orders of magnitude. For instance, on CIFAR-10, sharpness dropped from 27.7 to 0.0003.
  • Robustness: The method proved robust across various noise strengths and consistently provided a 1-2% accuracy boost even when combined with other state-of-the-art techniques like Ghost Batch Normalization.
Empirical analysis: (a) Sharpness visualization comparing small-batch (SB) and large-batch (LB) training landscapes, and (b) sensitivity of loss and accuracy to the noise parameter 'a'.

Conclusion

SmoothOut offers a principled way to improve generalization in deep learning by architectural smoothing of the loss landscape. By viewing noise injection as a signal processing tool, it provides a robust mechanism to escape sharp minima. While currently focused on CNNs for computer vision, the framework's success suggests broad applicability to other architectures like Transformers and RNNs, where generalization remains a critical challenge.

Summary updated

Abstract

In Deep Learning, Stochastic Gradient Descent (SGD) is usually selected as a training method because of its efficiency; however, recently, a problem in SGD gains research interest: sharp minima in Deep Neural Networks (DNNs) have poor generalization; especially, large-batch SGD tends to converge to sharp minima. It becomes an open question whether escaping sharp minima can improve the generalization. To answer this question, we propose SmoothOut framework to smooth out sharp minima in DNNs and thereby improve generalization. In a nutshell, SmoothOut perturbs multiple copies of the DNN by noise injection and averages these copies. Injecting noises to SGD is widely used in the literature, but SmoothOut differs in lots of ways: (1) a de-noising process is applied before parameter updating; (2) noise strength is adapted to filter norm; (3) an alternative interpretation on the advantage of noise injection, from the perspective of sharpness and generalization; (4) usage of uniform noise instead of Gaussian noise. We prove that SmoothOut can eliminate sharp minima. Training multiple DNN copies is inefficient, we further propose an unbiased stochastic SmoothOut which only introduces the overhead of noise injecting and de-noising per batch. An adaptive variant of SmoothOut, AdaSmoothOut, is also proposed to improve generalization. In a variety of experiments, SmoothOut and AdaSmoothOut consistently improve generalization in both small-batch and large-batch training on the top of state-of-the-art solutions.
BibTeX
@misc{wen2018smoothout,
  author = {Wei Wen and Yandan Wang and Feng Yan and Cong Xu and Yiran Chen and Hai Li},
  title = {{SmoothOut: Smoothing Out Sharp Minima for Generalization in Large-Batch Deep Learning}},
  year = {2018},
  eprint = {1805.07898},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.07898},
}
Figure