EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation

Shih-Yang Liu, Maksim Khadkevich, Nai Chit Fung, Charbel Sakr, Chao-Han Huck Yang, Chien-Yi Wang, Saurav Muralidharan, Hongxu Yin, Kwang-Ting Cheng, Jan Kautz, Yu-Chiang Frank Wang, Pavlo Molchanov, Min-Hung Chen

TL;DR

Post-training compression makes Large Language Models (LLMs) smaller and faster, but often at the cost of significant accuracy degradation. EoRA (Eigenspace Low-Rank Approximation) addresses this by adding lightweight, task-specific low-rank modules to compressed LLMs without any fine-tuning or backpropagation. Computed in just a few minutes using minimal calibration data, EoRA effectively recovers lost accuracy while offering a custom fused CUDA kernel to minimize inference latency overhead.

What Problem Does This Paper Address?

Deploying LLMs in production requires aggressive compression techniques like quantization (e.g., reducing weights to 3-bit or 4-bit integers) and pruning (e.g., 2:4 structured sparsity). While these methods successfully shrink memory footprints and speed up execution, they inevitably degrade model accuracy.

Furthermore, standard compression formats are rigidly constrained by hardware and kernel support. If a user wants a middle-ground trade-off between accuracy and efficiency, they cannot easily modify the underlying compressed format.

To bypass this, researchers use "customized compensation"—adding residual low-rank paths (like LoRA adapters) to a frozen, compressed backbone. However, existing approaches fall into two flawed camps:

  1. Naive SVD (Singular Value Decomposition): Analytical and fast, but it ignores the distribution of actual task data, leading to poor task-specific accuracy recovery.
  2. LoRA-based Fine-Tuning: Highly accurate, but requires expensive gradient-based training (backpropagation) that takes hours and substantial compute.

EoRA answers the crucial question: How can we swiftly improve task-specific accuracy for compressed LLMs within minutes, without fine-tuning?

Methods

EoRA introduces a fine-tuning-free mathematical approach that aligns low-rank error approximation directly with the model's actual layer-wise compression loss.

1. Eigenspace Projection

Instead of performing SVD directly on the weight error matrix ($\Delta W = W - \hat{W}$), EoRA projects the error into the eigenspace of the layer's input activations.

  • It computes the eigendecomposition of the average input activation matrix: $\tilde{X}\tilde{X}^T = Q\Lambda Q^T$, where $Q$ represents the eigenvectors and $\Lambda$ contains the eigenvalues.
  • The eigenvalues act as "importance scores," indicating which activation channels matter most.
  • The compression error is scaled and projected into this space: $\Delta W' = \Delta W Q\sqrt{\Lambda}$.
  • Standard SVD is then applied to this projected error matrix $\Delta W'$, ensuring that capacity is allocated to the most critical error elements. Finally, the low-rank matrices are projected back to the original space.

2. Dynamic Task Toggling

Because the base compressed LLM remains completely frozen, users can generate distinct, lightweight EoRA modules for different downstream tasks (e.g., math, coding, translation) and dynamically toggle them on or off at inference time using a single shared backbone.

Overview of the EoRA framework enabling dynamic task-specific modules

3. Fused CUDA Kernel

Adding low-rank paths introduces extra matrix multiplications, shifting inference from being computation-bound to memory-bound due to redundant L2 cache and DRAM transfers. To fix this, the authors designed a custom CUDA kernel that fuses the low-rank matrix multiplication ($B'$) directly with the weight quantization kernel, cutting down data movement.

Datasets

The authors evaluated EoRA across several diverse benchmarks using LLaMA2 (7B/13B) and LLaMA3 (8B/70B) models:

  • Language Generation: WikiText2 (evaluated via perplexity)
  • Commonsense Reasoning: ARC-Challenge (ARC-C)
  • Math Reasoning: MathQA and GSM8K
  • Summarization: CNN/DailyMail (ROUGE-Lsum)
  • Multi-task Language Understanding: MMLU

Calibration data was minimal, utilizing only 32 to 128 samples per task from datasets like WikiText2, ARC training sets, MathQA/GSM8K, and C4.

Key Insights and Ideas

  • Mathematical Alignment: The core "aha moment" is Theorem 1, which proves that performing SVD in the activation eigenspace is mathematically equivalent to minimizing the true layer-wise output discrepancy. This bridges the gap between fast analytical SVD and high-performing gradient descent.
  • Quantization Robustness: The low-rank matrices generated by EoRA can themselves be quantized down to 3-bit or 4-bit formats with negligible accuracy loss. In some cases, quantizing the EoRA matrices even acts as a regularizer, slightly improving performance.
  • LoRA Warm-Starting: If computational resources are available for fine-tuning later, EoRA serves as a vastly superior initialization method compared to standard QLoRA or LoftQ, accelerating convergence and reaching higher final accuracy.

Key Results

Sparsity & Quantization Recovery

When evaluating a LLaMA3-8B model pruned to 2:4 structured sparsity, EoRA (at rank 128) significantly outperformed other fine-tuning-free baselines (ZeroQuant-V2, Act-S) and matched or beat the training-based method (ApiQ):

  • ARC-C: Boosted accuracy from 30.11% (uncompensated) to 34.64%.
  • GSM8K: Rocketed accuracy from a broken 2.12% to 13.95%.

For a highly degraded 3-bit quantized LLaMA3-8B model, EoRA delivered massive accuracy leaps: +10.84% on ARC-Challenge, +6.74% on MathQA, and +11.45% on GSM8K.

As shown in the rank ablation plots, EoRA's performance scales exceptionally well as the rank increases from 64 to 512, widening the gap over other training-free methods:

Accuracy vs Rank across ARC-C, MathQA, and GSM8K

Efficiency and Speedups

  • Optimization Time: While gradient-based methods like ApiQ require roughly 2.5 hours of training, EoRA achieves comparable or superior results in under 15 minutes on a single NVIDIA H100 GPU.
  • Inference Speedup: The custom fused CUDA kernel delivers up to a 1.4× speedup over running low-rank compensation natively in PyTorch, successfully mitigating the latency overhead of the residual paths.

Limitations and Future Work

  • Residual Overhead: Even with the optimized fused kernel, adding low-rank paths introduces a slight latency penalty compared to a purely compressed model without any compensation.
  • Calibration Dependency: Although it requires very few samples (as few as 32-64), EoRA still depends on having representative downstream calibration data to construct the task-specific eigenspace.
  • Future Directions: The authors note potential extensions of the EoRA framework to non-transformer architectures and alternative modalities like vision or multi-modal models.

Summary updated

Abstract

While post-training compression techniques effectively reduce the memory footprint, latency, and power consumption of Large Language Models (LLMs), they often result in noticeable accuracy degradation and remain limited by hardware and kernel constraints that restrict supported compression formats - ultimately reducing flexibility across a wide range of deployment scenarios. In this work, we propose EoRA - a novel $\textbf{fine-tuning-free}$ method that augments compressed LLMs with low-rank matrices, allowing users to rapidly enhance task-specific performance and freely balance the trade-off between accuracy and computational overhead beyond the constraints of compression formats. EoRA consistently outperforms prior fine-tuning-free low rank methods in recovering the accuracy of compressed LLMs, achieving notable accuracy improvements (e.g., $\mathbf{10.84\%}$ on ARC-Challenge, $\mathbf{6.74\%}$ on MathQA, and $\mathbf{11.45\%}$ on GSM8K for LLaMA3-8B compressed to 3-bit). We also introduce an optimized CUDA kernel, accelerating inference by up to 1.4x and reducing memory overhead through quantizing EoRA. Overall, EoRA offers a prompt solution for improving the accuracy of compressed models under varying user requirements, enabling more efficient and flexible deployment of LLMs. Code is available at https://github.com/NVlabs/EoRA.
BibTeX
@inproceedings{liu2026eora,
  author = {Shih-Yang Liu and Maksim Khadkevich and Nai Chit Fung and Charbel Sakr and Chao-Han Huck Yang and Chien-Yi Wang and Saurav Muralidharan and Hongxu Yin and Kwang-Ting Cheng and Jan Kautz and Yu-Chiang Frank Wang and Pavlo Molchanov and Min-Hung Chen},
  title = {{EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation}},
  booktitle = {TTU Main Track @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=CBVNZfPSGs},
}
Figure