SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching

Tianyi Zhang, Junda Su, Oscar Wu, Zhaozhuo Xu, Anshumali Shrivastava

Introduction

Fine-tuning Large Language Models (LLMs) remains a resource-intensive challenge. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have gained popularity by adding small trainable adapters to quantized base models, they suffer from two primary limitations. First, they rely on a "low-rank" assumption for weight updates, which recent empirical evidence suggests may be overly restrictive. Second, they often require a "two-path" computation—processing both frozen base weights and adapters—which increases latency and memory overhead during training and inference.

Methodology: The SketchTune Framework

To address these bottlenecks, we introduce SketchTune, a novel framework that unifies model compression and adaptation through sketching. Unlike traditional methods that append parameters, SketchTune represents the entire model weight matrix as a compact, trainable "sketch."

The SketchTune architecture: Original weights are compressed into shared sketched parameters via a sketching matrix and mapping codes, allowing for direct optimization during fine-tuning.

The process involves several key innovations:

  1. Learned Sketching: Original weights are clustered into shared "sketched parameters" using a weighted k-means approach. This process is guided by the inverse Hessian diagonal to ensure that high-influence weights are preserved with greater fidelity.
  2. Unified Representation: Instead of adding an external adapter, the model weights are approximated on-the-fly from these sketches using a fixed mapping matrix ($M$).
  3. Direct Optimization: During fine-tuning, the algorithm updates the values within the sketches themselves. Since the mapping is differentiable, the model learns an optimal compressed representation tailored to the target task.
  4. Efficiency via Custom Kernels: To make this practical, the team developed custom CUDA kernels that handle weight reconstruction and gradient computation in a single path, significantly reducing the memory footprint and time-to-first-token (TTFT).

The High-Rank Reality of Weight Updates

A core motivation for SketchTune is the empirical observation that fine-tuned weight updates ($\Delta$) are often high-rank. Analysis of models like Llama-3-8B and Qwen2.5-7B reveals that capturing 75% of the variance in weight updates requires a rank exceeding 1000. Standard LoRA configurations, typically restricted to ranks between 8 and 64, are mathematically incapable of capturing this complexity. Sketching provides a more robust approximation that handles these high-rank patterns more effectively.

Experimental Results

SketchTune was evaluated across demanding benchmarks, including mathematical reasoning (GSM8K), commonsense reasoning, and instruction following (MT-Bench).

Efficiency and Latency

SketchTune demonstrates significant improvements in computational efficiency. Compared to LoRA and QLoRA, it achieves a 2.0–2.4× lower time-to-first-token (TTFT) and consumes 1.6–2.7× less GPU memory during training. These gains are largely attributed to the single-path computation enabled by its unified architecture.

Performance metrics across different batch sizes and context lengths, comparing LoRA, QLoRA, and SketchTune variants. SketchTune consistently shows lower latency and memory usage.

Accuracy and Compression

On math benchmarks, SketchTune consistently outperformed leading PEFT methods while maintaining a smaller footprint. It surpassed LoRA and DoRA while using base models that were 2.6–3.5× smaller. Notably, on the GSM8K benchmark, it exceeded LoftQ's accuracy by 14.48% despite using 7.3× fewer trainable parameters. In instruction-following tasks (MT-Bench), SketchTune achieved higher average scores than full fine-tuning, even with a model size reduction of 67.8%.

Conclusion

SketchTune represents a shift from additive adapters to unified compressive adaptation. By leveraging sketching to capture high-rank weight updates and optimizing a compressed representation directly, it achieves superior accuracy and efficiency compared to existing PEFT methods. While the approach requires a one-time preprocessing step for sketching, the resulting gains in latency and memory usage make it a compelling solution for deploying high-performance LLMs on resource-constrained hardware.

Summary updated

BibTeX
@inproceedings{zhang2025spallm,
  author = {Tianyi Zhang and Junda Su and Oscar Wu and Zhaozhuo Xu and Anshumali Shrivastava},
  title = {{SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=zZXOXhxO6I},
}
Figure