Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression

Alper Kalle, Theo Rudkiewicz, Mohamed-Oumar Ouerfelli, Mohamed TamaazoustiarXiv 2026 · arXiv

TL;DR

Complete-SVD is a novel post-training low-rank compression method for Large Language Models (LLMs) that minimizes the functional distance between the original and compressed models' outputs. By weighting all logit changes equally and using a clever "broadcast backward" estimation technique, it achieves superior generalization across diverse datasets compared to existing output-aware methods.

What Problem Does This Paper Address?

Deploying massive LLMs is prohibitively expensive due to their enormous parameter counts. Low-rank compression—replacing large weight matrices with products of two smaller ones—is a popular solution, but it often severely degrades model performance if not done carefully.

Traditional methods focus on minimizing error at the individual layer level (e.g., Frobenius norm of weight differences). More recent "activation-aware" methods improve this by considering the distribution of input activations. However, even these don't fully capture how a layer's error propagates through the rest of the network to the final output. Existing "output-aware" methods try to fix this but often focus too narrowly on preserving the probability of the specific next token observed in a small calibration set. This can lead to "overfitting" to the calibration data, causing the model to perform poorly on other types of text.

Methods

The authors introduce Complete-SVD, which aims to minimize the $L_2$ distance between the full logit vectors of the original and compressed models across the entire data distribution.

Key technical components include:

  • Functional Norm Objective: Instead of just looking at weights or layer outputs, it looks at the model's end-to-end behavior (the logits).
  • Linearization & K-FAC: To make the optimization tractable, they linearize the network around its original weights and use a Kronecker-factored approximation (K-FAC). This reduces the complex objective to a weighted Singular Value Decomposition (SVD) with a closed-form solution.
  • Tractable Estimation via "Broadcast Backward": Computing the output-side weight factor is normally impossible for LLMs because it scales with vocabulary size (often 32k–128k). The authors propose a "broadcast backward" estimator using random probes. This trick allows them to estimate the necessary statistics with a fixed number of backward passes, regardless of vocabulary size or sequence length.

They also compare this to KL-SVD, a variant that uses Kullback–Leibler divergence as the objective, which is more closely related to preserving the predictive distribution.

Figure 1: Output geometry changes the compression tradeoff. (a) The output-aware methods can be written in this Kronecker-factored form and differ mainly in the output kernel 𝒦︀: 𝐶input = Σinput, and 𝐶grad = Σgrad for Complete-SVD. (b) Illustrative next-token example. Fisher-based objectives compare 𝑝= softmax(𝑜), whereas Complete-SVD compares the logits 𝑜; local output costs hold to leading order in 𝛿𝑜. Link width is the weight of each token in this cost, and black ticks mark original values. Bars share a logarithmic scale (𝑜= log 𝑝+ const); compressed values are schematic. (c) Perplexity reduction 100(PPLKL −PPLComplete)/PPLKL of Complete- SVD relative to KL-SVD at compression ratios (CR) 0.9 and 0.7, with uniform ranks. WikiText-2 is the calibration domain.

Datasets

  • Calibration: WikiText-2 and C4 were used to collect activation and gradient statistics.
  • Evaluation (Perplexity): WikiText-2, C4, and PTB (Penn Treebank).
  • Downstream Benchmarks: A suite of zero-shot tasks including OpenbookQA, WinoGrande, HellaSwag, PIQA, ARC-challenge, and ARC-easy.
  • Instruction Following: The IFEval benchmark was used to test generative performance.

Key Insights and Ideas

  • Uniform Weighting Wins for Generalization: Fisher-based methods prioritize preserving the "correct" next token. While this works well on the calibration domain, Complete-SVD’s uniform weighting of all logits leads to much better performance on "out-of-distribution" corpora.
  • The Vocabulary Bottleneck is Breakable: The "broadcast backward" technique is a major practical contribution. It proves that we can perform sophisticated, output-aware compression on models with massive vocabularies without needing astronomical compute resources.
  • Synergy with Quantization: Low-rank compression and quantization are complementary. Complete-SVD can be used as a pre-processing step before weight quantization (like GPTQ), leading to extreme model size reductions with minimal accuracy loss.

Key Results

  • Superior Perplexity: On models like Mistral-7B, Llama 3-8B, and Qwen3-8B, Complete-SVD consistently achieved the lowest perplexity on C4 and PTB datasets across different compression ratios.
  • Significant Gains: For Llama 3-8B at a 0.9 compression ratio, it reduced PTB perplexity from 30.9 (the previous best baseline, IO-SVD) to 23.0.
  • Better Instruction Following: Complete-SVD outperformed prior output-aware methods on the IFEval benchmark, showing it preserves the model's ability to follow complex instructions better.
  • Extreme Compression: By combining Complete-SVD with 3-bit GPTQ quantization, the authors compressed Llama 2-7B from 13.5 GB down to just 1.9 GB (14% of original size) while maintaining a respectable perplexity of 16.7 on WikiText-2.

Limitations and Future Work

  • Scale: The study focused on models in the 7–8B parameter range. It remains to be seen if these advantages hold for even larger models (e.g., 70B+).
  • Error Accumulation: Like most post-training SVD methods, it compresses layers one by one. It doesn't account for how errors from previously compressed layers might compound, a challenge that "block-wise" refinement methods try to address at higher computational costs.
  • Beyond Low-Rank: The authors suggest that their functional norm approach could be adapted for other compression techniques like pruning or quantization itself, though this would require developing new dedicated solvers.

Summary updated

Abstract

Post-training low-rank compression of neural networks must not only approximate the original weights but also preserve the model's input-output behavior. Recent output-aware methods measure the compression error beyond the compressed layer, at the model output, weighting logit changes by the observed next token or by the model's predicted probabilities. We propose Complete-SVD, a method for large language models that minimizes the $L_2$ distance between the logits of the original and compressed models, weighting all logit changes equally. Linearizing the network around the original weights, together with a Kronecker-factored approximation, reduces this objective to a weighted SVD with a closed-form solution. Its output-side factor is intractable to compute; we estimate it without bias from random probes, at a cost of $K$ backward passes per sequence, independent of vocabulary size and sequence length. At matched settings with uniform ranks and WikiText-2 calibration, Complete-SVD achieves the lowest C4 and PTB perplexity in every setting, and at ratio 0.7 under LEMS rank allocation. Replacing the $L_2$ distance by the Kullback-Leibler divergence (KL-SVD), with the same estimation framework, lowers calibration perplexity in most settings and raises downstream accuracy, but raises cross-corpus perplexity. Complete-SVD also improves IFEval instruction-following accuracy over the strongest prior output-aware methods throughout.
BibTeX
@misc{kalle2026complete,
  author = {Alper Kalle and Theo Rudkiewicz and Mohamed-Oumar Ouerfelli and Mohamed Tamaazousti},
  title = {{Complete-SVD: A Functional Norm Approach to Low-Rank LLM Compression}},
  year = {2026},
  eprint = {2611.26059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26059},
}
Figure