Large-scale Transformer models like BERT and GPT-3 are notoriously redundant. Previous research has shown that you can often prune up to 90% of their parameters with only a marginal drop in performance. This raises a provocative question: Are these "redundant" parameters inherently useless, or are they simply victims of poor training strategies?
The authors of this paper argue that standard optimizers often favor a specific subset of parameters, providing them with sufficient training while leaving others under-fitted. This creates a "rich-get-richer" dynamic where a few weights do all the heavy lifting. Instead of just cutting out the "lazy" parameters through pruning, this paper proposes SAGE (Sensitivity-guided Adaptive learninG ratE)—a novel training strategy that forces every parameter to contribute effectively to the model's success.
The SAGE Approach
SAGE is an adaptive learning rate schedule guided by two main factors:
- Sensitivity ($I$): This is a robust, gradient-based measure that approximates how much the model's loss would change if a specific parameter were removed.
- Low Sensitivity: The parameter is currently redundant or under-fitted. SAGE increases its learning rate to encourage it to learn more.
- High Sensitivity: The parameter is already well-trained. SAGE decreases its learning rate to prevent it from overfitting.
- Local Temporal Variation ($U$): Because sensitivity is calculated on mini-batches, it can be noisy and unstable. $U$ measures the uncertainty of the sensitivity by comparing the current value to an exponential moving average. If uncertainty is high, SAGE avoids drastically slowing down the learning rate.
The resulting update rule is elegant: the learning rate for a parameter is proportional to the ratio of its uncertainty to its sensitivity ($\eta \propto U/I$). SAGE is designed to be a "plug-and-play" module that works seamlessly on top of existing optimizers like SGD, Adam, and Adamax.
Key Insights: Making Every Weight Count
The paper challenges the "Lottery Ticket Hypothesis" view that some weights are just "born" useless. Instead, it suggests that redundancy is often a result of the training process itself.
Balanced Contribution: Analysis shows that SAGE results in a more concentrated sensitivity distribution. In other words, the "workload" is more evenly distributed across all parameters compared to standard training.
The "Aha" Moment on Pruning: When the authors tried to prune a SAGE-trained model, performance dropped immediately. In contrast, pruning a standard model often improves performance initially. This proves that SAGE successfully turned "redundant" weights into "contributing" weights—there was no "fat" left to trim.
Furthermore, visualization on a toy "Spiral" dataset showed that SAGE produces smoother decision boundaries with larger margins compared to Adam, explaining its superior generalization capabilities.
Empirical Results
The researchers evaluated SAGE across three major domains, demonstrating consistent improvements:
- Natural Language Understanding (NLU): On the GLUE benchmark, SAGE improved the average score of BERT-base by 1.4 points and RoBERTa-large by 0.6 points. It was particularly effective on smaller datasets like RTE and CoLA, where it gained over 2 points.
- Neural Machine Translation (NMT): On the rich-resource WMT’16 En-De dataset using Transformer-base, SAGE improved the BLEU score by 0.4 points, showing it helps even when training from scratch rather than just fine-tuning.
- Computer Vision: For ViT-B/32 on ImageNet, SAGE boosted top-1 accuracy from 81.28% to 81.72%.
Notably, SAGE is complementary to other techniques. When combined with SMART (an adversarial regularization method), it achieved state-of-the-art results on the GLUE leaderboard for its model class.
Limitations and Future Work
While highly effective, SAGE does introduce a new hyperparameter, $\beta_0$ (the moving average coefficient for sensitivity), which represents an extra knob for researchers to tune. Additionally, while the computational cost is described as "marginal," SAGE requires storing and updating an exponential moving average for every single parameter, which increases memory usage.
The study focuses heavily on Transformers. While the results on Vision Transformers suggest broad applicability, further testing on other architectures like Graph Neural Networks (GNNs) or specialized CNNs remains an exciting avenue for future work.
Summary updated