AMC: Adaptive Learning Rate Adjustment Based on Model Complexity

Weiwei Cheng, Rong Pu, Bin WangMathematics 2025 · DOI

TL;DR

This paper introduces AMC, a novel optimizer that dynamically adjusts learning rates based on model complexity, addressing instability and slow convergence in deep learning. By using the Frobenius norm to quantify complexity, AMC reduces learning rates for complex models and increases them for simpler ones, leading to improved training stability and faster convergence.

What Problem Does This Paper Address?

Training deep learning models often involves optimizers that adapt learning rates based on gradient information. However, popular optimizers like Adam, while effective, don't explicitly consider the inherent complexity of the model being trained. This oversight can lead to issues, especially with complex models: learning rates that are too high can cause instability and oscillations, while rates that are too low can result in slow convergence. The paper highlights this by showing that a more complex VGG-19 model trained with Adam exhibits more training loss fluctuations than a simpler VGG-11 model, even with the same hyperparameters. The core problem is that a one-size-fits-all adaptive learning rate strategy, based solely on gradient statistics, can be suboptimal when model complexity varies significantly.

Methods

The paper proposes AMC (Adam with Model Complexity), an optimizer that extends Adam by incorporating model complexity into its learning rate adjustment mechanism.

  • Model Complexity Measurement: AMC uses the Frobenius norm of the model's parameter matrix ($||W||_F$) as a proxy for model complexity. This metric captures both the magnitude and distribution of parameters, offering a more nuanced view than simple parameter counts or layer depths. A larger Frobenius norm indicates a more complex model.
  • Dynamic Learning Rate Adjustment: The learning rate ($\eta_t$) is adjusted using the formula: $$ \eta_t = \frac{\eta_0}{1 + \gamma ||W||_F} $$ where $\eta_0$ is the initial learning rate and $\gamma$ is a hyperparameter controlling the sensitivity to model complexity. This formula ensures that as model complexity ($||W||_F$) increases, the learning rate decreases, promoting stability. Conversely, for simpler models with a small $||W||_F$, the learning rate increases, accelerating convergence.

  • Adam's Core Mechanism: AMC retains Adam's core functionality of computing first-order (momentum) and second-order (variance) moments of gradients and applying bias correction. The parameter update then uses this adjusted learning rate.

  • Theoretical Analysis: The paper provides theoretical analysis to demonstrate the relationship between model complexity and gradient fluctuations, proving that higher complexity leads to larger fluctuations. It also establishes convergence properties and bounds for AMC, showing it converges to a local minimum with a sublinear rate.

Datasets

The experiments were conducted on two classic image classification datasets:

  • MNIST: A dataset of handwritten digits (28x28 pixels).
  • Fashion-MNIST: A dataset of clothing images (28x28 pixels), serving as a slightly more challenging alternative to MNIST.

The paper uses these datasets to train and evaluate four VGG models with varying complexities: VGG-11, VGG-13, VGG-16, and VGG-19.

Key Insights and Ideas

  • Model Complexity as a First-Class Citizen: The most significant insight is treating model complexity not just as an outcome of architecture but as a direct input to the optimization process. This moves beyond purely gradient-based adaptation.
  • Frobenius Norm as a Robust Complexity Metric: Using the Frobenius norm is a clever way to capture the "effective" complexity of a model's parameters, accounting for their distribution and magnitude, which directly influences gradient behavior.
  • Balancing Stability and Speed: The core idea is that complex models need more cautious exploration (lower learning rate) for stability, while simpler models can afford aggressive exploration (higher learning rate) for speed. AMC elegantly balances these competing needs.
  • Mitigating Gradient Noise: The dynamic learning rate adjustment is shown to be effective in reducing oscillations and improving stability, particularly in scenarios with noisy gradients, which are common in deep learning.

Key Results

  • Improved Training Stability: AMC consistently achieved lower and smoother training loss curves across all tested VGG models (VGG-11 to VGG-19) on both MNIST and Fashion-MNIST datasets, compared to Adam, AdamW, and QHAdam. This indicates reduced oscillations and more stable training.
  • Faster Convergence: AMC demonstrated faster convergence, reaching lower training losses in fewer epochs, especially in the early stages of training.
  • Higher Accuracy: AMC achieved the highest testing accuracy across all models and datasets. For instance, on VGG-19 with MNIST, AMC reached 99.17% testing accuracy, outperforming Adam (99.02%), AdamW (99.01%), and QHAdam (98.97%).
  • Minimal Computational Overhead: The paper notes that the additional computational cost of calculating the Frobenius norm for AMC is minimal, making it practical for real-world applications.
  • Superior Performance on Complex Models: The benefits of AMC were particularly pronounced when training more complex models like VGG-16 and VGG-19, where it significantly outperformed other optimizers in terms of stability and accuracy.

Limitations and Future Work

  • Limited Architectures Tested: The authors acknowledge that while VGG models are representative, they plan to explore AMC's performance on other architectures like MLPs, encoder-decoder models, and LSTMs in future work.
  • Hyperparameter Tuning: While AMC introduces a new hyperparameter ($\gamma$), the paper implies it can be tuned to balance complexity sensitivity. Further research could explore automated methods for setting this.
  • Broader Domain Applicability: The authors suggest future work could investigate AMC's applicability in a wider range of domains and tasks beyond image classification.
  • Hybrid Optimization: Combining AMC with other advanced techniques like adaptive regularization is proposed as a direction to further enhance its flexibility and adaptability.

Summary updated

Abstract

An optimizer plays a decisive role in the efficiency and effectiveness of model training in deep learning. Although Adam and its variants are widely used, the impact of model complexity on training is not considered, which leads to instability or slow convergence when a complex model is trained. To address this issue, we propose an AMC (Adam with Model Complexity) optimizer, which dynamically adjusts the learning rate by incorporating model complexity, thereby improving training stability and convergence speed. AMC uses the Frobenius norm of the model to measure its complexity, automatically decreasing the learning rate of complex models and increasing the learning rate of simple models, thus optimizing the training process. We provide a theoretical analysis to demonstrate the relationship between model complexity and learning rate, as well as the convergence and convergence bounds of AMC. Experiments on multiple benchmark datasets show that, compared to several widely used optimizers, AMC exhibits better stability and faster convergence, especially in the training of complex models.
BibTeX
@inproceedings{cheng2025amc,
  author = {Weiwei Cheng and Rong Pu and Bin Wang},
  title = {{AMC: Adaptive Learning Rate Adjustment Based on Model Complexity}},
  booktitle = {Mathematics},
  year = {2025},
  doi = {10.3390/math13040650},
}
Figure