SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching

Tianyi Zhang, Junda Su, Oscar Wu, Zhaozhuo Xu, Anshumali ShrivastavaSubmission to ICLR 2025 · Rejected

TL;DR

Existing compressive adaptation methods like QLoRA use a clunky "two-tower" design that freezes a low-bit base model and tunes an additive low-rank adapter. SpaLLM ditches this split architecture by compressing weight rows into small lookup tables (codebooks) and directly fine-tuning the table entries themselves. By eliminating low-rank constraints, SpaLLM achieves full-rank updates, outperforms QLoRA and LoftQ across reasoning benchmarks, and delivers a 3× inference speedup.

What Problem Does This Paper Address?

Fine-tuning modern Large Language Models (LLMs) requires immense memory. While methods like QLoRA and LoftQ enable fine-tuning on consumer hardware by combining low-bit quantization with Parameter-Efficient Fine-Tuning (PEFT), they rely on a two-tower architecture: frozen low-precision weights operate alongside full-precision, low-rank additive adapters (like LoRA).

This paradigm suffers from two main shortcomings:

  1. The Low-Rank Bottleneck: These methods assume weight updates are low-rank. However, empirical studies show task-specific adaptation updates can be high-rank. When base weights are heavily compressed (e.g., down to 2 or 3 bits), low-rank adapters struggle to recover lost representational capacity.
  2. System & Inference Inefficiency: Managing two distinct computation paths—one running low-bit base weights and another running full-precision adapters—adds computational overhead, requires dequantization, and complicates serving multiple fine-tuned models.

Methods

SpaLLM proposes a unified, single-tower compression and adaptation workflow based on element-wise parameter sketching:

  • Row-Wise Sketching into Lookup Tables: For every weight row $\theta \in \mathbb{R}^d$, weights are approximated as $\hat{\theta} = \Pi w$, where $\Pi \in \mathbb{R}^{d \times k}$ is a row-wise one-hot mapping matrix (indices pointing to table entries) and $w \in \mathbb{R}^k$ represents a lookup table (LUT) of centroids.
  • Weighted Initialization: Centroids $w$ are learned using Lloyd’s algorithm, inversely weighted by the diagonal of the layer's input Hessian ($\text{diag}((X X^\top)^{-1})$). Quantization error is iteratively minimized using error-compensation updates inspired by GPTQ and LeanQuant.
  • Fine-Tuning the Tables (LUTs): During adaptation, the discrete assignment matrix $\Pi$ is frozen, and backpropagation directly updates only the floating-point values in $w$.
  • Granular Scaling via Groups Per Row (GPR): To tune parameter capacity without blowing up the codebook indices, rows are divided into contiguous groups (e.g., 1, 2, 4, or 8 groups per row), each maintaining its own independent sketch table.
  • Single-Tower Inference: Because the fine-tuned parameters are embedded directly in the lookup tables, inference requires only a single compressed matrix multiplication per layer—bypassing runtime dequantization and dual-path additions.
Two-Tower (QLoRA):     Input ──┬──> [Frozen Quantized Base] ──> (+) ──> Output
                               └──> [LoRA Adapter (A x B)]  ──┘

Single-Tower (SpaLLM):  Input ─────> [Sketched Weights (Fixed Codes + Tuned LUT)] ─────> Output

Datasets

  • Training Datasets: WikiText-2 (language modeling), GSM8K (grade-school math), and Stanford Alpaca (52k instruction-tuning samples).
  • Evaluation Benchmarks:
  • Perplexity on WikiText-2.
  • Accuracy on GSM8K.
  • Zero-shot accuracy on CommonsenseQA suites: PIQA, WinoGrande, HellaSwag, OpenBookQA, BoolQ, and ARC (Easy & Challenge).
  • LLM-as-a-judge evaluations (using GPT-4o) on instructions sampled from Alpaca and the Vicuna benchmark.
  • Base Models Tested: LLaMA-7B, LLaMA-2 (7B, 13B), and LLaMA-3 (8B, 70B).

Key Insights and Ideas

  • High-Rank Expressivity via Parameter Sharing: An identity matrix is full-rank and fundamentally cannot be approximated well with low-rank factorizations, yet it can be represented perfectly with just two shared values ($0$ and $1$). Parameter sharing allows high-rank or full-rank updates while remaining extremely parameter-efficient.
  • Compression as Implicit Regularization: Mapping multiple virtual weights to single trainable centroids prevents the model from overfitting to small downstream datasets, leading to better general reasoning retention.
  • Eliminating "Two-Tower" Tax: In standard PEFT, maintaining separate branches slows down memory-bound auto-regressive decoding. Merging adaptation directly into quantized representation kernels restores pure single-pass inference.

Key Results

  • WikiText-2 & GSM8K:
  • On LLaMA-2-13B at 3-bit compression, SpaLLM achieved a perplexity of 5.05, beating both 3-bit LoftQ (5.13) and 16-bit full-precision LoRA (5.12).
  • On GSM8K math reasoning, 4-bit SpaLLM (GPR=8) achieved 38.4% on LLaMA-2-7B (vs. 35.1% for QLoRA and 35.0% for LoftQ) and 49.1% on LLaMA-2-13B (beating 16-bit LoRA’s 43.1%).
  • At severe 2-bit compression on LLaMA-2-13B, SpaLLM reached 33.8% accuracy, while LoftQ scored 25.4% and QLoRA failed to converge.
  • Commonsense QA: On LLaMA-7B (4-bit), SpaLLM achieved a 65.8% average across 5 QA tasks using only 22 million trainable parameters, outperforming QLoRA variants that required ~160 million parameters.
  • LLaMA-3-70B at Scale: Fine-tuned on Alpaca at 4 bits (GPR=4), SpaLLM occupied just 39.8 GB (fitting on a single NVIDIA L40S GPU) and achieved a 72% average on 7 CommonsenseQA tasks, exceeding the 141 GB full-precision base model (71%). In GPT-4o judge head-to-heads, it achieved a 91% win-loss ratio against full-precision Falcon-40B-Instruct.
  • Inference Efficiency: Evaluating 128-token generation with specialized kernels showed SpaLLM delivers a ~3× speedup over QLoRA/LoftQ (e.g., 0.06s vs. 0.19s per token on LLaMA-2-7B) alongside lower peak VRAM.

Limitations and Future Work

  • Hardware-Specific Kernels: SpaLLM's decoding throughput gains rely on specialized lookup table multiplication kernels (such as those from SqueezeLLM), which are less ubiquitously supported across non-GPU edge hardware than standard linear GEMM kernels.
  • Fixed Assignment Grid: The codebook mapping $\Pi$ remains frozen after initial quantization; adapting the cluster assignments dynamically during fine-tuning was not explored.
  • Evaluation Scope: Experiments focused almost entirely on the LLaMA architecture family; extending this unified sketching paradigm to mixture-of-experts (MoE) or non-transformer architectures remains open for validation.

Summary updated

Abstract

Compressive adaptation approaches, such as QLoRA, are widely popular alternatives for reducing memory requirements during fine-tuning of large language models (LLMs) while producing models capable of handling various downstream tasks. The key idea is to employ a “two-tower” architecture: compressing pretrained LLM parameters into compact representations and fine-tuning the additive full-precision adapter, which typically has few tunable parameters in low-rank format. However, the strict algebraic assumptions, such as low-rank assumption, and the complexity of composing two-tower architectures are some of the known shortcomings, resulting in a poor accuracy-efficiency trade-off. In response to these known limitations, we propose SpaLLM (Sketched Parameter Adaptation of LLMs), a novel compressive adaptation approach for LLMs. This method is also the first to illustrate parameter-sharing compression methods for LLM fine-tuning, which, unlike QLoRA, are free from strict low-rank algebraic assumptions on adapters. Furthermore, our proposal unifies model compression and adaptation into a single, streamlined process, eliminating the need for two-tower architectures. SpaLLM sketches pre-trained LLM weights into lookup tables and directly fine-tunes the values in these tables. This approach simplifies LLMs’ compressive adaptation workflow, potentially improves multi-user serving efficiency, and delivers significantly better accuracy for both natural language understanding and generation tasks. Moreover, by avoiding the “two-tower” architecture, our framework only requires one compressed matrix multiplication per layer during inference, demonstrating superior inference efficiency compared to previous methods.
BibTeX
@inproceedings{zhang2025spallm,
  author = {Tianyi Zhang and Junda Su and Oscar Wu and Zhaozhuo Xu and Anshumali Shrivastava},
  title = {{SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=zZXOXhxO6I},
}
Figure