No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models

Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao

Large-scale Transformer models like BERT and GPT-3 are notoriously redundant. Previous research has shown that you can often prune up to 90% of their parameters with only a marginal drop in performance. This raises a provocative question: Are these "redundant" parameters inherently useless, or are they simply victims of poor training strategies?

The authors of this paper argue that standard optimizers often favor a specific subset of parameters, providing them with sufficient training while leaving others under-fitted. This creates a "rich-get-richer" dynamic where a few weights do all the heavy lifting. Instead of just cutting out the "lazy" parameters through pruning, this paper proposes SAGE (Sensitivity-guided Adaptive learninG ratE)—a novel training strategy that forces every parameter to contribute effectively to the model's success.

The SAGE Approach

SAGE is an adaptive learning rate schedule guided by two main factors:

  1. Sensitivity ($I$): This is a robust, gradient-based measure that approximates how much the model's loss would change if a specific parameter were removed.
  2. Low Sensitivity: The parameter is currently redundant or under-fitted. SAGE increases its learning rate to encourage it to learn more.
  3. High Sensitivity: The parameter is already well-trained. SAGE decreases its learning rate to prevent it from overfitting.
  4. Local Temporal Variation ($U$): Because sensitivity is calculated on mini-batches, it can be noisy and unstable. $U$ measures the uncertainty of the sensitivity by comparing the current value to an exponential moving average. If uncertainty is high, SAGE avoids drastically slowing down the learning rate.

The resulting update rule is elegant: the learning rate for a parameter is proportional to the ratio of its uncertainty to its sensitivity ($\eta \propto U/I$). SAGE is designed to be a "plug-and-play" module that works seamlessly on top of existing optimizers like SGD, Adam, and Adamax.

Key Insights: Making Every Weight Count

The paper challenges the "Lottery Ticket Hypothesis" view that some weights are just "born" useless. Instead, it suggests that redundancy is often a result of the training process itself.

Balanced Contribution: Analysis shows that SAGE results in a more concentrated sensitivity distribution. In other words, the "workload" is more evenly distributed across all parameters compared to standard training.

Distribution of parameter sensitivity on the MNLI dataset.

The "Aha" Moment on Pruning: When the authors tried to prune a SAGE-trained model, performance dropped immediately. In contrast, pruning a standard model often improves performance initially. This proves that SAGE successfully turned "redundant" weights into "contributing" weights—there was no "fat" left to trim.

Accuracy vs. Percentage of Weight Remaining on the RTE dataset.

Furthermore, visualization on a toy "Spiral" dataset showed that SAGE produces smoother decision boundaries with larger margins compared to Adam, explaining its superior generalization capabilities.

Empirical Results

The researchers evaluated SAGE across three major domains, demonstrating consistent improvements:

  • Natural Language Understanding (NLU): On the GLUE benchmark, SAGE improved the average score of BERT-base by 1.4 points and RoBERTa-large by 0.6 points. It was particularly effective on smaller datasets like RTE and CoLA, where it gained over 2 points.
  • Neural Machine Translation (NMT): On the rich-resource WMT’16 En-De dataset using Transformer-base, SAGE improved the BLEU score by 0.4 points, showing it helps even when training from scratch rather than just fine-tuning.
  • Computer Vision: For ViT-B/32 on ImageNet, SAGE boosted top-1 accuracy from 81.28% to 81.72%.

Notably, SAGE is complementary to other techniques. When combined with SMART (an adversarial regularization method), it achieved state-of-the-art results on the GLUE leaderboard for its model class.

Limitations and Future Work

While highly effective, SAGE does introduce a new hyperparameter, $\beta_0$ (the moving average coefficient for sensitivity), which represents an extra knob for researchers to tune. Additionally, while the computational cost is described as "marginal," SAGE requires storing and updating an exponential moving average for every single parameter, which increases memory usage.

The study focuses heavily on Transformers. While the results on Vision Transformers suggest broad applicability, further testing on other architectures like Graph Neural Networks (GNNs) or specialized CNNs remains an exciting avenue for future work.

Summary updated

Abstract

Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generalization performance. However, we question whether the redundant parameters could have contributed more if they were properly trained. To answer this question, we propose a novel training strategy that encourages all parameters to be trained sufficiently. Specifically, we adaptively adjust the learning rate for each parameter according to its sensitivity, a robust gradient-based measure reflecting this parameter's contribution to the model performance. A parameter with low sensitivity is redundant, and we improve its fitting by increasing its learning rate. In contrast, a parameter with high sensitivity is well-trained, and we regularize it by decreasing its learning rate to prevent further overfitting. We conduct extensive experiments on natural language understanding, neural machine translation, and image classification to demonstrate the effectiveness of the proposed schedule. Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
BibTeX
@inproceedings{liang2022no,
  author = {Chen Liang and Haoming Jiang and Simiao Zuo and Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen and Tuo Zhao},
  title = {{No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=nuj0RF43rWm},
}
Figure