No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models

Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, Tuo Zhao

TL;DR

SAGE (Sensitivity-guided Adaptive learninG ratE) is a novel training strategy that eliminates parameter redundancy in large Transformers by adaptively adjusting per-parameter learning rates. By accelerating the training of "under-fitted" parameters and regularizing "well-fitted" ones, SAGE significantly boosts generalization performance across natural language understanding, machine translation, and image classification tasks.

What Problem Does This Paper Address?

Modern Transformer models are massive, often containing hundreds of millions or even billions of parameters. However, research consistently shows that these models are highly redundant—you can often prune up to 90% of the weights with only a marginal drop in performance. In fact, as shown in Figure 1, pruning a standard BERT-base model by 20% can actually improve its accuracy, suggesting that redundant parameters aren't just "useless," they might actively hurt generalization.

Figure 1: Validation results of fine-tuning BERT-base at different sparsity levels on the RTE dataset (Wang et al., 2018) in Liang et al. (2021). Solid black curve represents the full model performance.

The authors of this paper ask a provocative question: Are these parameters truly redundant, or are they just victims of poor training strategies? They hypothesize that standard optimizers (like Adam or SGD) tend to favor certain parameters, leaving others under-trained and effectively "dead." Instead of throwing these parameters away through pruning, the authors propose a way to make them contribute to the model's success.

Methods

The authors introduce SAGE, an adaptive learning rate schedule that operates at the individual parameter level. The core idea is to guide each parameter's learning pace based on its sensitivity—a measure of how much that specific parameter contributes to the model's loss.

  1. Sensitivity ($I_j$): Calculated as the magnitude of the gradient-weight product. A parameter with high sensitivity is "important" (well-fitted), while low sensitivity indicates redundancy (under-fitted).
  2. Local Temporal Variation ($U_j$): Because sensitivity can be noisy due to mini-batch sampling, SAGE tracks the uncertainty of the sensitivity measure. If a parameter's sensitivity is fluctuating wildly, the optimizer treats its "redundancy" status as unreliable.
  3. The SAGE Schedule: The learning rate for each parameter is adjusted by a modulating term: $$\eta_j^{(t)} \propto \frac{\text{Uncertainty}_j^{(t)}}{\text{Sensitivity}_j^{(t)}}$$
    • Low Sensitivity? The learning rate increases to "excite" the parameter and encourage better fitting.
  4. High Sensitivity? The learning rate decreases, acting as a form of regularization to prevent the parameter from overfitting.

SAGE is designed to be a "plug-and-play" addition to existing optimizers like Adam, Adamax, or SGD.

Datasets

The authors evaluated SAGE across three major domains:

  • Natural Language Understanding (NLU): The GLUE benchmark (including tasks like MNLI, QNLI, SST-2, and RTE) using BERT-base and RoBERTa-large.
  • Neural Machine Translation (NMT): IWSLT’14 De-En (low-resource) and WMT’16 En-De (rich-resource) using Transformer-base.
  • Image Classification: CIFAR100 and ImageNet using Vision Transformer (ViT) models.

Key Insights and Ideas

  • Balanced Training: SAGE forces the model to distribute the "workload" more evenly across its parameters. As seen in Figure 2, the sensitivity distribution in SAGE-optimized models is much more concentrated compared to standard Adam. This means fewer parameters are left behind in a "useless" state.
Figure 2: The sensitivity distribution of the BERT-base models fine-tuned on GLUE tasks. Note that we drop some outliers to ease visualization.
  • Redundancy as a Choice: The paper challenges the "Lottery Ticket Hypothesis" (which suggests some weights are just "born" better). SAGE demonstrates that with the right training schedule, "bad" weights can be trained into "good" ones.
  • Better Decision Boundaries: Qualitative analysis on synthetic datasets (Figure 5) shows that SAGE produces smoother decision boundaries with larger margins, a classic hallmark of superior generalization.
  • Robustness to Noise: By incorporating "Local Temporal Variation," SAGE avoids prematurely penalizing important parameters whose gradients are temporarily unstable, making the training process more robust.

Key Results

  • GLUE Benchmark: SAGE improved the average score of BERT-base by 1.4 points and RoBERTa-large by 0.6 points. The gains were particularly pronounced on smaller datasets like RTE (+9.8% for BERT-base).
  • Machine Translation: SAGE achieved a +0.6 BLEU improvement on IWSLT and +0.4 BLEU on WMT, proving it helps when training from scratch, not just during fine-tuning.
  • Vision Transformers: On ImageNet, SAGE boosted ViT-B/32 accuracy from 81.28% to 81.72%.
  • Pruning Resistance: In SAGE-trained models, pruning actually hurts performance immediately (see Figure 3). This is a good sign—it means the "redundant" parameters were actually doing useful work and were no longer safe to remove.
Figure 3: Unlike the baseline, pruning SAGE-optimized models consistently leads to performance drops, showing that "redundant" weights have become useful.

Limitations and Future Work

  • Memory Overhead: SAGE requires storing an exponential moving average of the sensitivity for every parameter, which increases the memory footprint during training.
  • Hyperparameter Tuning: While SAGE is robust, it introduces a new hyperparameter ($\beta_0$ for the moving average) that may require some tuning for optimal results.
  • Scaling: While tested on models up to RoBERTa-large, the authors suggest future work could explore applying SAGE to even larger "foundation" models (like GPT-3 scale) where redundancy is even more prevalent.

Summary updated

Abstract

Recent research has shown the existence of significant redundancy in large Transformer models. One can prune the redundant parameters without significantly sacrificing the generalization performance. However, we question whether the redundant parameters could have contributed more if they were properly trained. To answer this question, we propose a novel training strategy that encourages all parameters to be trained sufficiently. Specifically, we adaptively adjust the learning rate for each parameter according to its sensitivity, a robust gradient-based measure reflecting this parameter's contribution to the model performance. A parameter with low sensitivity is redundant, and we improve its fitting by increasing its learning rate. In contrast, a parameter with high sensitivity is well-trained, and we regularize it by decreasing its learning rate to prevent further overfitting. We conduct extensive experiments on natural language understanding, neural machine translation, and image classification to demonstrate the effectiveness of the proposed schedule. Analysis shows that the proposed schedule indeed reduces the redundancy and improves generalization performance.
BibTeX
@inproceedings{liang2022no,
  author = {Chen Liang and Haoming Jiang and Simiao Zuo and Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen and Tuo Zhao},
  title = {{No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=nuj0RF43rWm},
}
Figure