Abstract
Matrix decomposition methods, such as Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformer-based language models. While importance-weighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two fundamental assumptions: 1) unchanged importance distribution during further fine-tuning, 2) equal importance across weight matrices in different layers. Furthermore, these methods necessitate a well-trained task-specific model as the starting point and require additional fine-tuning after compression. In this work, we proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process. Starting from a general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuning. The extensive evaluations demonstrate that RankDyna can outperform current SOTA methods under various parameter budget levels, and the advantage of RankDyna is further enhanced with higher compression rates.
Introduction
Transformer-based models like BERT, GPT, and BART have revolutionized natural language processing but come with significant computational and memory costs. Deploying these models on resource-constrained devices necessitates effective compression techniques. Matrix factorization, particularly Singular Value Decomposition (SVD), is a popular approach that decomposes large weight matrices into smaller, low-rank components.
However, traditional low-rank methods face three primary challenges. First, they often assume static importance, where the significance of specific parameters is fixed throughout training. Second, they typically employ uniform allocation, assigning the same rank budget to every layer regardless of its sensitivity. Finally, state-of-the-art importance-weighted methods often require a computationally expensive multi-stage pipeline: pre-fine-tuning, compression, and post-compression fine-tuning.
RankDyna: Dynamic Matrix Decomposition
RankDyna addresses these limitations by integrating compression directly into the fine-tuning process. Instead of a post-processing step, RankDyna allows the model to adapt to a task and shrink simultaneously.
Key Components
- Singular Groups: RankDyna treats a singular value and its corresponding vectors as a "singular group," which serves as the fundamental unit for rank allocation.
- Dynamic Importance Tracking: Using a first-order Taylor expansion and a gradient-based approach, the algorithm estimates the impact of removing a singular group on the training loss. A momentum scheme is used to smooth these scores, capturing the evolving importance of parameters as the model converges.
- Global Rank Allocation: RankDyna evaluates singular groups across the entire model globally. By keeping only the top-N most important groups, the model dynamically discovers which layers and weight matrices require more capacity.
- Cubic Scheduler: The model begins at full rank and gradually reduces to the target compression ratio using a cubic decay scheduler, pruning the least important groups at each step.
Key Insights and Analysis
A core finding of this work is that parameter importance is fluid. As shown in Figure 3, the relative importance of singular groups shifts significantly during the fine-tuning process. Static methods that calculate importance only at the start or end of training miss this evolution, leading to sub-optimal pruning.
Furthermore, the global allocation strategy reveals a non-uniform distribution of critical parameters. RankDyna naturally "front-loads" capacity, allocating more rank to the initial layers—particularly the Attention matrices ($W_q, W_k, W_v$)—while aggressively pruning the later Feed-Forward Network (FFN) layers.
Experimental Results
RankDyna was evaluated across various tasks, including the GLUE benchmark for NLU, SAMSUM for summarization, and Wikitext2/PTB for language modeling.
- BERT on GLUE: At a high compression rate (13.9M linear parameters), RankDyna achieved an average score of 80.4, significantly outperforming the previous best importance-weighted method (TFWSVD at 77.8) and standard SVD (71.7).
- Extreme Compression: When pushed to just 6.4M parameters, RankDyna maintained a score of 76.7, whereas standard SVD performance collapsed to 60.8.
- Generative Performance: On the PTB dataset with GPT-2, RankDyna achieved a perplexity of 39.4, nearly doubling the efficiency of competing methods (next best at 62.9).
Conclusion
RankDyna represents a shift from static to dynamic model compression. By unifying adaptation and compression into a single training round and globally allocating rank based on real-time importance, it achieves superior performance with less overhead. While it introduces some training-time memory overhead to track importance scores, the resulting compact models are highly efficient for inference on edge devices.
Summary updated