Transformers have revolutionized machine learning but their massive parameter counts make them difficult to deploy on edge devices. While pruning—the removal of redundant weights—is a standard solution for Convolutional Neural Networks (CNNs), traditional methods often fail when applied to Transformers. This paper introduces SEVEN (REserve SEntinels), a noise-robust pruning framework designed specifically for the complex gradient dynamics of Transformer Models (TMs).
The Problem: Stochastic Gradient Noise
The authors identify Stochastic Gradient Noise (SGN) as the primary obstacle to efficient Transformer pruning. In CNNs, gradient importance is relatively stable. In contrast, Transformers exhibit high Relative Gradient Variation (RGV). A weight might show a massive gradient spike in one training batch due to data noise, only to become irrelevant in the next.
Existing methods like SNIP or GraSP often fall into a "noise trap," selecting Temporary Sentinel Weights (TSW) that appear important in a single snapshot but do not contribute to long-term model performance. The challenge lies in identifying true Sentinel Weights (SW)—parameters that maintain stable sensitivity over time.
Methodology: The SEVEN Framework
SEVEN introduces a scoring function designed to act as a "low-pass filter" for gradient noise, ensuring only the most robust weights are preserved. The method relies on three core components:
- Polyak Averaging: Instead of single-shot snapshots, SEVEN tracks the first and second moments of gradients (similar to the Adam optimizer). This allows the algorithm to estimate both the mean importance and the variance (noise) of each weight.
- Correction Coefficients: Inspired by Symbolic Descent, SEVEN applies a correction factor that suppresses weights with high variance and highlights those with stable, high-magnitude gradients.
- Synapse Scoring: The final importance score is an accumulation of these corrected gradients over multiple iterations, ensuring the selection of weights with consistent sensitivity.
SEVEN is implemented in two modes: SEVEN$_{pre}$, a pre-pruning approach to find "lottery tickets" before training, and SEVEN$_{dyn}$, a dynamic approach that iteratively removes weights during fine-tuning.
Key Results and Performance
SEVEN was evaluated across diverse benchmarks, including GLUE (NLP) and ImageNet (Vision). The results demonstrate that stability of sensitivity is a far better predictor of weight importance than raw magnitude in Transformers.
- Vision Tasks: On CIFAR-100 at 80% sparsity, SEVEN$_{pre}$ achieved a 3.21% accuracy improvement over the next best method.
- Language Tasks: SEVEN$_{dyn}$ frequently outperformed the original unpruned models. For instance, on the STS-B dataset, the pruned model achieved a score of 90.17, surpassing the full BERT-BASE score of 88.4.
- Robustness: The authors found that SEVEN is so effective at initial weight selection that "weight resurrection" (re-enabling pruned weights) provides negligible benefit, simplifying the overall pruning pipeline.
Limitations and Future Outlook
While SEVEN sets a new state-of-the-art for unstructured pruning, several areas remain for future exploration. Currently, the method focuses on individual weight removal; extending SEVEN to structured pruning (removing entire neurons or heads) would be necessary for immediate hardware acceleration on standard GPUs. Additionally, while the method scales to BERT and ViT, its application to modern Large Language Models (LLMs) like Llama remains to be tested, where gradient noise may be even more significant.
Summary updated