AdaNorm: Adaptive Gradient Norm Correction Based Optimizer for CNNs

Shiv Ram Dubey, Satish Kumar Singh, Bidyut Baran ChaudhuriWACV 2023

TL;DR

This paper introduces AdaNorm, a novel optimizer that enhances existing gradient-based methods by adaptively correcting the norm of gradients. By ensuring gradients are more consistent with historical trends, AdaNorm improves training stability and performance for Convolutional Neural Networks (CNNs) across various datasets and architectures.

What Problem Does This Paper Address?

Training deep neural networks, particularly CNNs, relies heavily on stochastic gradient descent (SGD) and its adaptive variants like Adam, RMSProp, and AdaBelief. While these optimizers have significantly advanced the field, they often struggle with inconsistent or "bad" gradients, especially in flat regions of the loss landscape or due to the inherent noise in mini-batch computations. This inconsistency can lead to slow convergence, oscillations, and suboptimal performance. Existing adaptive methods primarily focus on adapting the learning rate based on the magnitude of past gradients but don't explicitly address the norm of the current gradient in relation to its historical behavior. This paper aims to bridge this gap by introducing a mechanism to correct the gradient norm, making it more representative and stable throughout training.

Methods

The core innovation of this paper is the AdaNorm approach, which adaptively corrects the norm of the gradient in each iteration. This is achieved by:

  • Historical Gradient Norm Estimation: AdaNorm maintains an Exponential Moving Average (EMA) of the gradient norm from past iterations. This EMA, denoted by $e_t$, captures the historical trend of gradient magnitudes.
  • Gradient Norm Correction: For the current gradient $g_t$, its L2 norm is calculated as $g_{norm} = ||g_t||_2$. If the historical gradient norm ($e_t$) is greater than the current gradient norm ($g_{norm}$), the current gradient $g_t$ is scaled up by the ratio $e_t/g_{norm}$ to ensure its norm is at least the historical norm. This scaled gradient is denoted as $s_t$. Otherwise, the original gradient $g_t$ is used.
  • Integration with Existing Optimizers: The proposed AdaNorm is designed to be generic and can be integrated with any existing adaptive SGD optimizer. The paper demonstrates this by applying AdaNorm to Adam, diffGrad, Radam, and AdaBelief, creating new optimizers like AdamNorm, diffGradNorm, RadamNorm, and AdaBeliefNorm. Specifically, for AdamNorm, the corrected gradient $s_t$ is used to update the first moment ($m_t$), while the original gradient $g_t$ is used for the second moment ($v_t$) to avoid negatively impacting the learning rate adaptation.

Datasets

The paper evaluates the proposed AdaNorm optimizers on three standard computer vision datasets:

  • CIFAR10: A dataset with 10 object classes.
  • CIFAR100: A dataset with 100 object classes, considered more fine-grained than CIFAR10.
  • TinyImageNet: A subset of ImageNet with 200 object classes, known for its larger scale and complexity.

Key Insights and Ideas

  • Gradient Norm as a Crucial Signal: The paper highlights that not just the direction but also the magnitude (norm) of gradients carries important historical information. Inconsistent gradient norms can hinder convergence.
  • Boosting Gradients in Flat Regions: AdaNorm's core mechanism of scaling up gradients when their norm is below the historical average is particularly effective in flat regions of the loss landscape, where gradients are typically small and can lead to slow progress.
  • Generic Applicability: The design of AdaNorm as a plug-and-play module for existing optimizers is a significant advantage, allowing researchers to easily experiment with its benefits on their preferred optimizers.
  • Improved Stability and Performance: By ensuring a more consistent and representative gradient norm, AdaNorm leads to more stable training and improved accuracy across various CNN architectures and datasets.

Key Results

  • Accuracy Improvements: AdaNorm consistently improved classification accuracy across all tested CNN models (VGG16, ResNet18, ResNet50) and datasets (CIFAR10, CIFAR100, TinyImageNet) when integrated with Adam, diffGrad, Radam, and AdaBelief. For instance, on TinyImageNet with ResNet50, AdaNorm achieved an 11.15% improvement over Adam.
  • Enhanced Gradient Norm: Figure 3 demonstrates that AdaNorm maintains a significantly higher and more representative mean gradient norm throughout training compared to its base optimizer (e.g., Adam).
  • Faster Convergence (in some cases): While Adam might converge faster initially on some datasets, AdaNorm shows better overall performance and avoids saturation, especially on CIFAR100 and TinyImageNet, due to more consistent updates.
  • Robustness to Batch Size: The paper shows that AdaNorm is less sensitive to batch size variations compared to Adam, likely due to the gradient norm correction mitigating the impact of noisy mini-batches.

Limitations and Future Work

  • Hyperparameter Tuning: The performance of AdaNorm is influenced by the hyperparameter $\gamma$, which controls the EMA of the gradient norm. The optimal value of $\gamma$ can vary across datasets, with smaller values being more suitable for larger datasets like TinyImageNet.
  • Impact on Second Moment: The paper found that applying gradient norm correction to the second moment (which controls the learning rate) can be detrimental, leading to poorer performance. This suggests that the correction is best applied only to the first moment.
  • Future Work: The authors suggest further exploration of the impact of AdaNorm on different network architectures and tasks. They also implicitly suggest investigating more sophisticated ways to incorporate historical gradient information beyond EMA. The paper also mentions that the convergence rate in terms of regret is similar to Adam, but empirical results show better practical performance, hinting at a gap between theoretical bounds and real-world effectiveness that could be further explored.

Summary updated

BibTeX
@inproceedings{dubey2023adanorm,
  author = {Shiv Ram Dubey and Satish Kumar Singh and Bidyut Baran Chaudhuri},
  title = {{AdaNorm: Adaptive Gradient Norm Correction Based Optimizer for CNNs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure