Sketch-Based Low-Rank Model Merging with Shared Circulant Transforms

Zhiming Zhang, Rong Yin, Xiaoshuai Hao, Hansong Zhang, Hao Peng, Yong Liu, Can Ma, Dan MengICML 2026 · OpenReview

TL;DR

CircuMerge is a new framework that slashes the time required to merge multiple LoRA adapters by over 44% without sacrificing accuracy. By applying a shared circulant transform to "spread out" the model's energy before sketching it down to a smaller size, it allows researchers to merge complex models in a fraction of the time required by traditional dense or subspace-based methods.

What Problem Does This Paper Address?

Low-Rank Adaptation (LoRA) has become the gold standard for fine-tuning large models efficiently. However, as we move toward "multi-task" ecosystems, we often end up with dozens of specialized adapters for different tasks (e.g., one for sentiment analysis, one for coding, one for medical text).

The challenge arises when we want to merge these adapters into a single, multi-talented model. Current methods face a "latency bottleneck":

  1. Dense Merging: Treating LoRA updates as full-sized weight matrices is computationally expensive and memory-intensive.
  2. Subspace Merging: Methods that try to find a shared "core" space often rely on Singular Value Decomposition (SVD), which is notoriously slow when dealing with many layers and tasks.

The authors identify that we need a way to summarize these adapters (sketching) that is fast, preserves task-specific information, and avoids the heavy lifting of matrix factorization.

Methods

The authors propose CircuMerge, a pipeline that operates in five distinct steps:

  1. Shared Circulant Transform (SCT): Before doing anything else, every adapter is passed through a shared, invertible circulant matrix. Because circulant matrices are tied to the Fast Fourier Transform (FFT), this step is incredibly fast ($O(r \log r)$).
  2. Sketching: Once the adapters are in this new "Fourier-aligned" coordinate system, the authors simply sample a small subset of rows and columns. This creates a "sketch"—a tiny version of the adapter that still captures the essential interactions.
  3. Merge in Sketch Space: The actual merging logic (like Task Arithmetic, TIES, or DARE) is performed on these tiny sketches rather than the full matrices.
  4. Reconstruction: The merged sketch is placed back into a full-sized matrix. The parts that weren't sampled are filled in using simple element-wise averaging.
  5. Recovery: The shared circulant transform is inverted to bring the merged adapter back to the standard LoRA format.

The "secret sauce" here is the circulant transform. It acts as a "spectral uniformizer," spreading the important information (singular value energy) evenly across the matrix so that random sampling doesn't accidentally throw away vital task data.

Datasets

The authors evaluated CircuMerge across two major domains:

  • Language: Using Llama 3 8B adapters on a 6-task Natural Language Inference (NLI) benchmark (including SNLI, MNLI, SICK, QNLI, RTE, and SciTail).
  • Vision: Using CLIP (ViT-B/32 and ViT-L/14) encoders on an 8-task suite (Cars196, DTD, EuroSAT, GTSRB, MNIST, RESISC45, SUN397, and SVHN).

Key Insights and Ideas

  • Spectral Uniformization: The most profound insight is the theoretical link between "effective rank" and merging quality. If a matrix's energy is concentrated in just a few directions, sketching it is risky. By using a circulant transform to "flatten" the spectrum, the authors make the matrix more robust to sampling.
  • Alignment is Everything: By using a shared transform across all tasks, the authors force all adapters into a common coordinate system. This ensures that when you sample the 5th row of Task A and the 5th row of Task B, you are looking at comparable features.
  • Complexity Gains: CircuMerge shifts the heavy lifting from $O(n^3)$ (cubic time) to $O(nTr \log r)$. In plain English: as models get wider ($n$) and you add more tasks ($T$), CircuMerge stays fast while other methods crawl to a halt.

Key Results

  • Speed: CircuMerge reduces per-layer merging time by at least 44% compared to state-of-the-art methods like KnOTS or Core Space Merging. In some configurations, it is orders of magnitude faster (e.g., merging in 4-5 seconds vs. nearly an hour for KnOTS).
  • Accuracy: Despite working with compressed "sketches," CircuMerge matches or exceeds the accuracy of merging in the "Full" parameter space. On the Llama 3 NLI benchmark, it achieved SOTA-level results across multiple merging rules.
  • Robustness: The method proved effective even in "joint-task" settings where the model doesn't know which task it's performing at inference time, showing that the sketches preserve enough nuance to prevent task interference.

Limitations and Future Work

  • Theoretical Scope: While the empirical results cover many merging rules (TIES, DARE, TSV), the formal theoretical proof in the paper is currently scoped specifically to Task Arithmetic (TA).
  • Sketching Strategies: The current version uses uniform random sampling after the transform. The authors suggest that more "principled" or adaptive sketching strategies could further improve results.
  • Scaling Up: While tested on 8B parameter models, the behavior on "Ultra-Large" models (70B+) or much larger collections of tasks (hundreds of adapters) remains an open area for evaluation.

Summary updated

Abstract

Merging multiple low-rank adapters (LoRA) provides a practical route to scaling multi-task learning and deployment more efficiently than full-model weight merging, while avoiding reliance on task-specific training data. However, most existing approaches either treat LoRA updates as dense weight deltas or depend on expensive subspace factorizations, making the merge step a primary latency bottleneck. To address this issue, this paper establishes a theoretically positive relationship between merging quality and the effective rank of the matrices being merged. Motivated by this insight, we propose **CircuMerge**, a sketch-based framework for low-rank model merging built on shared circulant transforms. Especially, this approach treats each adapter as a pair of low-rank matrices and applies a shared circulant transform to align all tasks in a common coordinate system. This alignment enables more efficient sampling, allowing us to generate compact sketches that effectively summarize the interactions between tasks. These compact sketches enable applying the merging rules directly to them and reconstructing a standard low-rank adapter, preserving the essential information while significantly reducing computational overhead. Across a broad multi-task LoRA benchmarks covering both vision and language settings, extensive empirical results demonstrate that CircuMerge reduces the overall merging time by at least 44\% compared to the state-of-the-art approaches, with accuracy matching or exceeding the optimal level.
BibTeX
@inproceedings{zhang2026sketch,
  author = {Zhiming Zhang and Rong Yin and Xiaoshuai Hao and Hansong Zhang and Hao Peng and Yong Liu and Can Ma and Dan Meng},
  title = {{Sketch-Based Low-Rank Model Merging with Shared Circulant Transforms}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=ytlRCCl7Pf},
}
Figure