Deforming the Loss Surface

Liangming Chen, Long Jin, Xiujuan Du, Shuai Li, Liu Mei

TL;DR

This paper introduces a novel "deformation operator" to dynamically alter the shape of a deep learning model's loss surface during training. By making the loss surface "flatter" around minima, this technique helps optimizers find more generalizable solutions, leading to improved accuracy on standard image classification tasks with minimal computational overhead.

What Problem Does This Paper Address?

Deep learning models often struggle with generalization, meaning they perform well on training data but poorly on unseen data. A key hypothesis in the field is that "flat" minima in the loss landscape lead to better generalization compared to "sharp" minima. However, standard optimization algorithms can get stuck in these sharp minima, hindering performance. This paper tackles the challenge of guiding optimizers towards these desirable flat minima by actively reshaping the loss landscape itself.

Methods

The core innovation is the "deformation operator," specifically a "deformation function" that modifies the loss value based on its current magnitude. This function is applied during the optimization process. The authors propose two types of deformation functions: arctan-power-based (AP) and log-exp-based (LE). They also provide theoretical analysis showing that their deformation functions can help stochastic gradient descent (SGD) filter out sharp minima and converge to flatter regions. The proposed method is integrated into existing first-order optimization algorithms, requiring only a small computational overhead.

Datasets

The paper uses standard image classification datasets for evaluation:

  • CIFAR-10: A dataset of 60,000 images (32x32 pixels) across 10 classes.
  • CIFAR-100: A dataset of 60,000 images (32x32 pixels) across 100 classes.
  • ImageNet: A large-scale dataset with 1000 classes, used for more extensive experiments.

Key Insights and Ideas

  • Loss surface is not fixed: The paper challenges the common assumption that the loss surface is static during training, proposing that it can be actively manipulated.
  • Deformation as a generalization tool: By deforming the loss surface, the authors demonstrate a direct link to improving generalization performance, not just finding lower training loss.
  • Flat minima filter: The theoretical analysis suggests that their deformation approach acts as a "flat minima filter," guiding the optimizer away from sharp, poorly generalizing minima.
  • Minimal computational cost: A significant advantage is that these improvements are achieved with negligible additional computational overhead, making it practical for real-world applications.

Key Results

  • Improved Accuracy: On CIFAR-10, CIFAR-100, and ImageNet, popular CNN architectures (ResNet, PreResNet, DenseNet, EfficientNet, SE-ResNeXt) enhanced with deformation functions consistently outperformed their original counterparts.
  • Significant Gains: For example, ResNet-20 on CIFAR-100 saw a 1.46% increase in top-1 test accuracy. EfficientNet-B0 on CIFAR-100 improved by 1.19%.
  • Visual Evidence: Loss curve visualizations show that deformation functions lead to flatter minima compared to the original models.
  • No Additional Tricks: The improvements are achieved without introducing complex architectural changes or additional training tricks.

Limitations and Future Work

  • Focus on First-Order Methods: The current work primarily focuses on first-order optimization methods like SGD. It's not explicitly stated if the deformation can be applied to second-order methods.
  • Deformation Function Design: While AP and LE functions are proposed and analyzed, the paper suggests that other forms of deformation operators could be explored in the future.
  • Broader Applications: The authors mention that future work could investigate addressing other optimization challenges like saddle points and gradient explosion problems using deformation operators.
  • Computational Cost Trade-off: While the overhead is minimal, the paper acknowledges that the "broader impact" could involve slightly increased unemployment in the short term as automation becomes more efficient.

Summary updated

Abstract

In deep learning, it is usually assumed that the shape of the loss surface is fixed. Differently, a novel concept of deformation operator is first proposed in this paper to deform the loss surface, thereby improving the optimization. Deformation function, as a type of deformation operator, can improve the generalization performance. Moreover, various deformation functions are designed, and their contributions to the loss surface are further provided. Then, the original stochastic gradient descent optimizer is theoretically proved to be a flat minima filter that owns the talent to filter out the sharp minima. Furthermore, the flatter minima could be obtained by exploiting the proposed deformation functions, which is verified on CIFAR-100, with visualizations of loss landscapes near the critical points obtained by both the original optimizer and optimizer enhanced by deformation functions. The experimental results show that deformation functions do find flatter regions. Moreover, on ImageNet, CIFAR-10, and CIFAR-100, popular convolutional neural networks enhanced by deformation functions are compared with the corresponding original models, where significant improvements are observed on all of the involved models equipped with deformation functions. For example, the top-1 test accuracy of ResNet-20 on CIFAR-100 increases by 1.46%, with insignificant additional computational overhead.
BibTeX
@misc{chen2020deforming,
  author = {Liangming Chen and Long Jin and Xiujuan Du and Shuai Li and Liu Mei},
  title = {{Deforming the Loss Surface}},
  year = {2020},
  eprint = {2007.12515},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.12515},
}
Figure