The Unified Balance Theory of Second-Moment Exponential Scaling Optimizers in Visual Tasks

Gongyue Zhang, Honghai LiuarXiv 2024 · DOI · arXiv

Abstract

The choice between Stochastic Gradient Descent (SGD) and adaptive optimizers like Adam often presents a trade-off between generalization and convergence speed. This paper introduces the Unified Balance Theory, a framework that unifies these optimizers through a single "balance coefficient" ($\alpha$). By scaling the second-moment of gradients, we demonstrate that SGD and Adam are specific points on a continuous spectrum. Our findings suggest that for visual tasks, adjusting $\alpha$ to negative values can mitigate gradient imbalances inherent in backpropagation, leading to superior generalization compared to standard SGD.

Introduction

In the current deep learning landscape, a dichotomy exists between optimizers used for Computer Vision (CV) and those used for Large Language Models (LLMs). While SGD remains the standard for CV due to its superior generalization, Adam is preferred for LLMs for its rapid convergence. This "optimizer divide" is often attributed to the inherent imbalances in backpropagation, such as gradient vanishing and explosion, which cause different layers to train at disparate rates. Traditionally, these issues are addressed through architectural modifications like Batch Normalization or Residual connections. This work explores whether these imbalances can be rectified directly within the optimizer's update rule.

The Unified Balance Theory

We propose the Second-Moment Exponential Scaling (SMES) framework. The standard update rule is generalized as follows:

$$\theta_{t+1} = \theta_t - \frac{\eta}{(\hat{v}_t)^\alpha + \epsilon} \cdot \hat{m}_t$$

In this formulation, $\eta$ represents the learning rate, $\hat{m}_t$ the first-moment estimate, and $\hat{v}_t$ the second-moment estimate. The critical parameter is the balance coefficient ($\alpha$), which defines the optimizer's behavior:

  • $\alpha = 0$: The denominator simplifies to 1, recovering the standard SGD update.
  • $\alpha = 0.5$: The update rule corresponds to Adam.

By treating $\alpha$ as a continuous variable, we can navigate the spectrum between fitting (positive $\alpha$) and generalization (negative $\alpha$).

Experimental Methodology

To evaluate the impact of the balance coefficient, we conducted experiments on the CIFAR-10 and CIFAR-100 datasets. We utilized the VGG network family (VGG13, VGG16, and VGG19) as our primary testbeds. These architectures were chosen specifically because their simple, non-residual structures provide a clear view of how gradient imbalances affect training without the interference of skip connections.

Results and Discussion

Our empirical results indicate that the optimal optimizer configuration for visual tasks often lies outside the standard SGD and Adam definitions. Specifically, we observed that negative values of $\alpha$—representing a shift toward greater generalization—consistently yielded lower test errors.

Test Error vs. Balance Coefficient ($\alpha$) for VGG architectures on CIFAR-10. The vertical line at $\alpha=0$ represents standard SGD.

As illustrated in the figure above, the test error for all VGG variants decreases as $\alpha$ moves into the negative range, reaching an optimal point before rising again. For instance, on CIFAR-100 with VGG13, the optimal performance was achieved at $\alpha = -0.08$, significantly outperforming the $\alpha = 0$ (SGD) baseline.

This phenomenon supports the concept of "Training the Input End." By utilizing a negative balance coefficient, the optimizer suppresses excessive fitting near the output layers and encourages the network to develop more robust, fundamental visual features at the input layers. This effectively counters the "network isomerism" where different layers exhibit varying tendencies toward gradient stability.

Conclusion

The Unified Balance Theory provides a novel lens through which to view optimization. By introducing the balance coefficient, researchers gain a precise tool to tune the trade-off between a model's ability to fit training data and its ability to generalize to unseen examples. While this study focused on VGG architectures and CIFAR datasets, the results suggest a promising path for optimizing more complex models. Future work will involve extending this framework to modern architectures like Transformers and larger-scale datasets like ImageNet to further validate the universality of the balance coefficient.

Summary updated

BibTeX
@misc{zhang2024unified,
  author = {Gongyue Zhang and Honghai Liu},
  title = {{The Unified Balance Theory of Second-Moment Exponential Scaling Optimizers in Visual Tasks}},
  year = {2024},
  eprint = {2405.18498},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.18498},
}
Figure