Abstract: CURing is a high-speed model compression technique that leverages CUR matrix decomposition to shrink Large Language Models (LLMs) efficiently. By approximating weight matrices using original rows and columns and "healing" the resulting performance loss through subspace-constrained fine-tuning, CURing achieves significant size reduction (e.g., -9% for Llama 3.1-8B) with minimal degradation in accuracy.
Introduction
As Large Language Models (LLMs) scale, their deployment in resource-constrained environments becomes increasingly difficult. Traditional compression methods like pruning and Singular Value Decomposition (SVD) often require extensive retraining, take hours to process, and destroy the interpretability of original weight parameters. CURing addresses these issues by providing a deterministic, fast, and representation-preserving alternative.
Methodology
CURing approximates a weight matrix $W$ as the product of three matrices: $C$ (selected columns), $R$ (selected rows), and a small linking matrix $U$. Unlike SVD, which creates abstract singular vectors, CURing uses actual slices of the original model weights.
The process involves three main stages:
- Selection: Informative rows and columns are chosen using the WANDA metric, which combines weight magnitude and input activations, alongside the Discrete Empirical Interpolation Method (DEIM).
- Layer Selection: Layers are prioritized for compression based on "angular distance," identifying those that contribute least to data transformation.
- Subspace-Constrained Healing: To recover accuracy, the model undergoes Parameter-Efficient Fine-Tuning (PEFT) where $C$ and $R$ remain fixed, and only the linking matrix $U$ is updated ($U = U_0 + \Delta U$). This restricts updates to the original weight subspaces, preventing catastrophic forgetting.
Evaluation and Results
The authors evaluated CURing on Llama 3.1-8B using datasets like C4 and WikiText2 for perplexity, and benchmarks like BoolQ and MMLU for zero-shot and few-shot performance. A novel UUID-to-UUID mapping task was also used to test adaptation to new information.
CURing demonstrated exceptional speed, compressing Llama 3.1-8B in just 129 seconds on a single H100 GPU—orders of magnitude faster than methods like SliceGPT or LLM Surgeon. While initial compression increases perplexity, the healing phase restores performance rapidly. On the C4 dataset, the healed model achieved a perplexity of 17.56, surpassing the original baseline of 23.79. Furthermore, CURing showed superior stability compared to LoRA and MoRA when adapting to new tasks.
Key Insights and Limitations
By preserving the "soul" of the weights through $C$ and $R$ selection, CURing maintains the internal representation characteristics of the original model. The subspace constraint acts as a natural regularizer, allowing the model to heal without drifting into forgetting.
However, CURing is slightly slower at learning entirely new tasks compared to LoRA or MoRA because its updates are strictly constrained to the pre-selected subspace. Future work will explore extreme compression ratios (50%+) and more advanced decomposition techniques like Compact Matrix Decomposition (CMD).
Summary updated