CURing Large Models: Compression via CUR Decomposition

Sanghyeon Park, Soo-Mook MoonarXiv 2025 · DOI · arXiv

Abstract: CURing is a high-speed model compression technique that leverages CUR matrix decomposition to shrink Large Language Models (LLMs) efficiently. By approximating weight matrices using original rows and columns and "healing" the resulting performance loss through subspace-constrained fine-tuning, CURing achieves significant size reduction (e.g., -9% for Llama 3.1-8B) with minimal degradation in accuracy.

Introduction

As Large Language Models (LLMs) scale, their deployment in resource-constrained environments becomes increasingly difficult. Traditional compression methods like pruning and Singular Value Decomposition (SVD) often require extensive retraining, take hours to process, and destroy the interpretability of original weight parameters. CURing addresses these issues by providing a deterministic, fast, and representation-preserving alternative.

Methodology

CURing approximates a weight matrix $W$ as the product of three matrices: $C$ (selected columns), $R$ (selected rows), and a small linking matrix $U$. Unlike SVD, which creates abstract singular vectors, CURing uses actual slices of the original model weights.

Comparison of CURing architecture with LoRA and MoRA

The process involves three main stages:

  1. Selection: Informative rows and columns are chosen using the WANDA metric, which combines weight magnitude and input activations, alongside the Discrete Empirical Interpolation Method (DEIM).
Importance scoring matrix S used for row/column selection
  1. Layer Selection: Layers are prioritized for compression based on "angular distance," identifying those that contribute least to data transformation.
  2. Subspace-Constrained Healing: To recover accuracy, the model undergoes Parameter-Efficient Fine-Tuning (PEFT) where $C$ and $R$ remain fixed, and only the linking matrix $U$ is updated ($U = U_0 + \Delta U$). This restricts updates to the original weight subspaces, preventing catastrophic forgetting.

Evaluation and Results

The authors evaluated CURing on Llama 3.1-8B using datasets like C4 and WikiText2 for perplexity, and benchmarks like BoolQ and MMLU for zero-shot and few-shot performance. A novel UUID-to-UUID mapping task was also used to test adaptation to new information.

CURing demonstrated exceptional speed, compressing Llama 3.1-8B in just 129 seconds on a single H100 GPU—orders of magnitude faster than methods like SliceGPT or LLM Surgeon. While initial compression increases perplexity, the healing phase restores performance rapidly. On the C4 dataset, the healed model achieved a perplexity of 17.56, surpassing the original baseline of 23.79. Furthermore, CURing showed superior stability compared to LoRA and MoRA when adapting to new tasks.

Key Insights and Limitations

By preserving the "soul" of the weights through $C$ and $R$ selection, CURing maintains the internal representation characteristics of the original model. The subspace constraint acts as a natural regularizer, allowing the model to heal without drifting into forgetting.

However, CURing is slightly slower at learning entirely new tasks compared to LoRA or MoRA because its updates are strictly constrained to the pre-selected subspace. Future work will explore extreme compression ratios (50%+) and more advanced decomposition techniques like Compact Matrix Decomposition (CMD).

Summary updated

Abstract

Large deep learning models have achieved remarkable success but are resource-intensive, posing challenges such as memory usage. We introduce CURing, a novel model compression method based on CUR matrix decomposition, which approximates weight matrices as the product of selected columns (C) and rows (R), and a small linking matrix (U). We apply this decomposition to weights chosen based on the combined influence of their magnitudes and activations. By identifying and retaining informative rows and columns, CURing significantly reduces model size with minimal performance loss. For example, it reduces Llama3.1-8B's parameters to 7.32B (-9%) in just 129 seconds, over 20 times faster than prior compression methods.
BibTeX
@misc{park2025curing,
  author = {Sanghyeon Park and Soo-Mook Moon},
  title = {{CURing Large Models: Compression via CUR Decomposition}},
  year = {2025},
  eprint = {2501.04211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.04211},
}
Figure