EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

Shih-Yang Liu, Huck Yang, Chein-Yi Wang, Nai Chit Fung, Hongxu Yin, Charbel Sakr, Saurav Muralidharan, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, Min-Hung Chen

TL;DR

This paper introduces EoRA, a novel, fine-tuning-free method that significantly boosts the accuracy of compressed Large Language Models (LLMs) by adding lightweight, task-specific low-rank modules. EoRA achieves this by projecting compression errors into the "eigenspace" of activations, allowing it to intelligently prioritize error correction and outperform existing methods without requiring extensive training.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but also computationally expensive, making their deployment challenging. Compression techniques like quantization and pruning reduce their size and latency, but often at the cost of accuracy. Existing methods struggle to recover this lost accuracy, especially when dealing with diverse deployment needs or when models are aggressively compressed. Furthermore, hardware and kernel constraints limit the flexibility of compression formats, preventing users from fine-tuning the accuracy-latency trade-off. This paper tackles the problem of how to efficiently and flexibly enhance the task-specific accuracy of already compressed LLMs without the need for costly fine-tuning.

Methods

The core of EoRA lies in its "Eigenspace Low-Rank Approximation." Instead of directly approximating the compression error (the difference between the original and compressed weights), EoRA first projects this error into the eigenspace of the layer's input activations. This projection, guided by eigenvalues (which indicate the importance of different activation channels), allows EoRA to focus its low-rank approximation on the most critical error components. This approach ensures that the approximation error is directly aligned with the task-specific compression loss. Crucially, EoRA is "fine-tuning-free," meaning it doesn't require backpropagation or extensive training, allowing for rapid adaptation in minutes using minimal calibration data. The paper also introduces an optimized CUDA kernel that fuses low-rank operations with quantization, further accelerating inference.

Datasets

The authors evaluated EoRA on several standard LLM benchmarks:

  • WikiText2: Used for language generation tasks.
  • ARC-Challenge (ARC-C): Used for commonsense reasoning.
  • MathQA: Used for math reasoning.
  • CNN/DailyMail: Used for LLM summarization.

The calibration datasets were constructed by sampling sentences from these tasks, with specific details on sentence length and concatenation provided in the paper.

Key Insights and Ideas

  • Eigenspace Projection is Key: By projecting compression errors into the eigenspace of activations, EoRA can intelligently prioritize which errors to fix, leading to more effective accuracy recovery than methods that treat all errors equally.
  • Fine-tuning-Free is a Game-Changer: The ability to achieve significant accuracy improvements in minutes without retraining or backpropagation is a major advantage for rapid deployment and adaptation.
  • Flexibility Beyond Compression Formats: EoRA allows users to fine-tune accuracy and overhead beyond the limitations of fixed quantization or sparsity formats, offering a more adaptable solution.
  • Robustness to Aggressive Compression: EoRA shows strong performance even when models are heavily compressed (e.g., 3-bit quantization), where other methods struggle.
  • Synergy with Quantization: EoRA itself can be quantized, further reducing its memory footprint without significant accuracy loss, making it even more practical.

Key Results

EoRA consistently outperformed existing fine-tuning-free methods across various tasks and compression scenarios. For example:

  • On LLaMA3-8B pruned to 2:4 sparsity, EoRA achieved improvements of 4.53% on ARC-C, 3.48% on MathQA, and 11.83% on GSM8K compared to a baseline.
  • When compensating for 3-bit quantized LLaMA3-8B, EoRA provided 10.8%, 6.7%, and 11.5% improvement on ARC-C, MathQA, and GSM8K, respectively, over baselines.
  • The optimized CUDA kernel achieved inference speedups of up to 1.4x.
  • EoRA demonstrated robustness to different ranks and calibration dataset sizes.

Limitations and Future Work

  • Computational Overhead: While fine-tuning-free, EoRA still introduces some computational overhead and memory usage due to the added low-rank modules. The fused kernel and quantization of EoRA aim to mitigate this.
  • Initialization for Fine-tuning: While EoRA can be used as an initialization for LoRA fine-tuning, achieving state-of-the-art results still requires some fine-tuning. The paper suggests that EoRA provides a strong starting point.
  • Future Work: The authors mention potential extensions to new architectures and modalities, suggesting that the core principles of EoRA could be applied more broadly. They also highlight the potential for EoRA to serve as a strong initialization for further fine-tuning, which could be explored more deeply.

Summary updated

Abstract

In this work, we re-formulate the model compression problem into the customized compensation problem: Given a compressed model, we aim to introduce residual low-rank paths to compensate for compression errors under customized requirements from users (e.g., tasks, compression ratios), resulting in greater flexibility in balancing accuracy and overhead(inference and model size) without being bound to fixed compression formats. However, naively applying SVD to derive residual paths causes suboptimal utilization of the low-rank representation capacity. Instead, we propose Training-free Eigenspace Low-Rank Approximation (EoRA), a method that directly minimizes compression-induced errors without requiring gradient-based training, achieving fast optimization in minutes using a small amount of calibration data. EoRA projects compression errors into the eigenspace of input activations, leveraging eigenvalues to effectively prioritize the reconstruction of high-importance error components. Moreover, EoRA can be seamlessly integrated with fine-tuning and quantization to further improve effectiveness and efficiency. EoRA consistently outperforms previous methods in compensating errors for compressed LLaMA2/3 models on various tasks, such as language generation, commonsense reasoning, and math reasoning tasks (e.g., 31.31%/12.88% and 9.69% improvements on ARC-Easy/ARC-Challenge and MathQA when compensating LLaMA3-8B that is quantized to 4-bit and pruned to 2:4 sparsity). EoRA offers a scalable, training-free solution to compensate for compression errors, making it a powerful tool to deploy LLMs more flexibly. Code is available at https://github.com/NVlabs/EoRA.
BibTeX
@inproceedings{liu2026eora,
  author = {Shih-Yang Liu and Huck Yang and Chein-Yi Wang and Nai Chit Fung and Hongxu Yin and Charbel Sakr and Saurav Muralidharan and Kwang-Ting Cheng and Jan Kautz and Yu-Chiang Frank Wang and Pavlo Molchanov and Min-Hung Chen},
  title = {{EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation}},
  booktitle = {TTU Main Track @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=CBVNZfPSGs},
}
Figure