SEVEN: Pruning Transformer Model by Reserving Sentinels

Jinying Xiao, Ping Li, Jie Nie, Zhe Tang

Transformers have revolutionized machine learning but their massive parameter counts make them difficult to deploy on edge devices. While pruning—the removal of redundant weights—is a standard solution for Convolutional Neural Networks (CNNs), traditional methods often fail when applied to Transformers. This paper introduces SEVEN (REserve SEntinels), a noise-robust pruning framework designed specifically for the complex gradient dynamics of Transformer Models (TMs).

The Problem: Stochastic Gradient Noise

The authors identify Stochastic Gradient Noise (SGN) as the primary obstacle to efficient Transformer pruning. In CNNs, gradient importance is relatively stable. In contrast, Transformers exhibit high Relative Gradient Variation (RGV). A weight might show a massive gradient spike in one training batch due to data noise, only to become irrelevant in the next.

Existing methods like SNIP or GraSP often fall into a "noise trap," selecting Temporary Sentinel Weights (TSW) that appear important in a single snapshot but do not contribute to long-term model performance. The challenge lies in identifying true Sentinel Weights (SW)—parameters that maintain stable sensitivity over time.

Visualization of weight importance landscapes for different pruning methods on BERT-BASE (MRPC). SEVEN identifies more stable "sentinel" weights compared to the noisier distributions of SNIP and GraSP.

Methodology: The SEVEN Framework

SEVEN introduces a scoring function designed to act as a "low-pass filter" for gradient noise, ensuring only the most robust weights are preserved. The method relies on three core components:

  1. Polyak Averaging: Instead of single-shot snapshots, SEVEN tracks the first and second moments of gradients (similar to the Adam optimizer). This allows the algorithm to estimate both the mean importance and the variance (noise) of each weight.
  2. Correction Coefficients: Inspired by Symbolic Descent, SEVEN applies a correction factor that suppresses weights with high variance and highlights those with stable, high-magnitude gradients.
  3. Synapse Scoring: The final importance score is an accumulation of these corrected gradients over multiple iterations, ensuring the selection of weights with consistent sensitivity.

SEVEN is implemented in two modes: SEVEN$_{pre}$, a pre-pruning approach to find "lottery tickets" before training, and SEVEN$_{dyn}$, a dynamic approach that iteratively removes weights during fine-tuning.

Key Results and Performance

SEVEN was evaluated across diverse benchmarks, including GLUE (NLP) and ImageNet (Vision). The results demonstrate that stability of sensitivity is a far better predictor of weight importance than raw magnitude in Transformers.

Accuracy vs. Sparsity on MRPC and SST-2 datasets. SEVEN$_{pre}$ consistently outperforms Magnitude-based, SNIP, and GraSP methods, often staying above the dense baseline even at 50-70% sparsity.
  • Vision Tasks: On CIFAR-100 at 80% sparsity, SEVEN$_{pre}$ achieved a 3.21% accuracy improvement over the next best method.
  • Language Tasks: SEVEN$_{dyn}$ frequently outperformed the original unpruned models. For instance, on the STS-B dataset, the pruned model achieved a score of 90.17, surpassing the full BERT-BASE score of 88.4.
  • Robustness: The authors found that SEVEN is so effective at initial weight selection that "weight resurrection" (re-enabling pruned weights) provides negligible benefit, simplifying the overall pruning pipeline.

Limitations and Future Outlook

While SEVEN sets a new state-of-the-art for unstructured pruning, several areas remain for future exploration. Currently, the method focuses on individual weight removal; extending SEVEN to structured pruning (removing entire neurons or heads) would be necessary for immediate hardware acceleration on standard GPUs. Additionally, while the method scales to BERT and ViT, its application to modern Large Language Models (LLMs) like Llama remains to be tested, where gradient noise may be even more significant.

Summary updated

Abstract

Large-scale Transformer models (TM) have demonstrated outstanding performance across various tasks. However, their considerable parameter size restricts their applicability, particularly on mobile devices. Due to the dynamic and intricate nature of gradients on TM compared to Convolutional Neural Networks, commonly used pruning methods tend to retain weights with larger gradient noise. This results in pruned models that are sensitive to sparsity and datasets, exhibiting suboptimal performance. Symbolic Descent (SD) is a general approach for training and fine-tuning TM. In this paper, we attempt to describe the noisy batch gradient sequences on TM through the cumulative process of SD. We utilize this design to dynamically assess the importance scores of weights.SEVEN is introduced by us, which particularly favors weights with consistently high sensitivity, i.e., weights with small gradient noise. These weights are tended to be preserved by SEVEN. Extensive experiments on various TM in natural language, question-answering, and image classification domains are conducted to validate the effectiveness of SEVEN. The results demonstrate significant improvements of SEVEN in multiple pruning scenarios and across different sparsity levels. Additionally, SEVEN exhibits robust performance under various fine-tuning strategies. The code is publicly available at https://github.com/xiaojinying/SEVEN.
BibTeX
@inproceedings{xiao2024seven,
  author = {Jinying Xiao and Ping Li and Jie Nie and Zhe Tang},
  title = {{SEVEN: Pruning Transformer Model by Reserving Sentinels}},
  booktitle = {IJCNN},
  year = {2024},
}
Figure