A Spectral Framework for Universal Proxy Regularization in Model Merging

Shangge Liu, Danyue Qian, Qi Fan, Yinghuan Shi, Wenbin Li, Lei Wang, Yang Gao, Dacheng TaoarXiv 2026 · arXiv

TL;DR

This paper introduces a Spectral Framework to explain how task-agnostic "proxy" data (like ImageNet) can prevent parameter interference during model merging. By regularizing fine-tuning with a universal proxy, researchers can create "merge-friendly" models that achieve near-oracle performance without ever needing access to the private data of other tasks.

What Problem Does This Paper Address?

Model merging—specifically Task Arithmetic—is a powerful way to combine independently fine-tuned "expert" models into a single multi-task model without expensive joint training. However, it often fails due to parameter interference: the weight updates that make a model good at Task A might inadvertently degrade its performance on Task B.

To fix this, existing "pre-merging" methods constrain fine-tuning using data from the other tasks you plan to merge with. This creates a catch-22: the whole point of model merging is to keep tasks independent and private, but these fixes require sharing data between them.

The authors ask: Can we replace specific peer-task data with a universal, task-agnostic proxy distribution? And more importantly, why and when does such a proxy work?

Universal proxy distribution regularization concept.

Methods

The authors propose a Spectral Framework for Universal Proxy Regularization. They characterize the relationship between a proxy distribution ($Q$) and a target task ($P_t$) using two key spectral parameters:

  • Spectral Coverage ($\alpha$): Measures how well the proxy's gradient space spans the target task's space.
  • Spectral Leakage ($\beta$): Quantifies the gradient energy in directions the proxy fails to cover.
Overview of the Spectral Framework and the three instantiations.

They derive a theoretical bound proving that smaller $(\alpha, \beta)$ values lead to lower parameter interference. To put this into practice, they augment the fine-tuning objective with a proxy regularizer ($R_Q$) and study three ways to implement it:

  1. (J) Jacobian: Exact evaluation using Jacobian-vector products (accurate but expensive).
  2. (F) Frozen Fisher: Uses a pre-computed Fisher Information Matrix snapshot (efficient but can drift).
  3. (O) Output-drift: Directly penalizes changes in the model's functional output on proxy samples (efficient and robust).

Crucially, they analyze these methods under non-linear fine-tuning, moving beyond the simplified linear assumptions (NTK) common in previous literature.

Datasets

The framework was validated across both vision and language modalities:

  • Vision: CLIP models (ViT-B/32, B/16, L/14) evaluated on 8 benchmarks including Cars, DTD, EuroSAT, and MNIST.
  • Language: Llama-3.2 (1B, 3B) on the TRACE benchmark and Llama-3 (8B) on GLUE tasks using LoRA.
  • Proxies: Common general-purpose datasets like ImageNet (for vision) and WikiText-103 (for language) served as universal proxies.

Key Insights and Ideas

  • Broad Coverage Trumps Semantic Similarity: A proxy doesn't need to "look like" the target task. ImageNet works as a proxy for medical imagery or satellite data because it provides broad spectral coverage of the model's parameter space, not because the classes are similar.
  • The Three-Way Hierarchy: In real-world non-linear training, the three implementation methods behave differently. The Jacobian (J) method is the gold standard for accuracy, while the Output-drift (O) method is a "best of both worlds" choice—it's computationally cheap (just a forward pass) but its accuracy scales with the quality of the proxy.
  • The Natural Image Manifold: Even synthetic images (from Stable Diffusion) work significantly better than random noise as proxies. This is because they activate the "natural image manifold" that the pre-trained model was originally built on.
Comparison of the three instantiations and their behavior as proxy quality degrades.

Key Results

  • Performance Boost: On ViT-B/32, adding proxy regularization to standard fine-tuning lifted merged absolute accuracy by 6.7–13.4 percentage points.
  • Closing the Oracle Gap: The method consistently approached the performance of "oracle" methods that have illegal access to peer-task data. For example, on ViT-L/14, the proxy-based method reached 91.64% accuracy, nearly matching the oracle's 90.80%.
  • LLM Success: Using WikiText-103 as a proxy for Llama-3.2 models improved multi-task merging scores by up to 2 points without ever seeing the specific downstream task data during the regularization phase.
  • Efficiency: The Output-drift (O) instantiation was found to be the most favorable for high-quality proxies, being ~43% faster and using significantly less memory than Jacobian-based methods.

Limitations and Future Work

  • Proxy Construction: While existing datasets like ImageNet work well, the authors suggest future work could involve actively constructing optimal proxies that minimize the $(\alpha, \beta)$ criteria for specific architectures.
  • Beyond Additive Merging: The current framework is tailored for Task Arithmetic (additive merging). Extending these spectral insights to other merging paradigms, like SVD-based merging or model breadcrumbing, remains an open challenge.
  • Cross-Modal Merging: The paper treats vision and language separately; exploring universal proxies for cross-modal model composition is a promising next step.

Summary updated

Abstract

Model merging constructs multi-task models without expensive joint training but often suffers from parameter interference. Some solutions mitigate this by constraining finetuning using data from the other tasks to be merged, violating the core independence and privacy assumption of model merging. Recent work offers preliminary empirical evidence that a universal, task-agnostic proxy distribution can substitute for such data, but leaves open when and why it suffices. To this end, we propose a \textit{Spectral Framework for Universal Proxy Regularization}, which characterizes the relationship between a proxy distribution and the distribution of other tasks to be merged through spectral coverage ($α$) and leakage ($β$). Based on the framework, we derive a unified parameter interference bound: smaller $(α, β)$ provably yields lower parameter interference through better spectral alignment. To exploit such a proxy, we augment the training objective with a proxy regularizer. Evaluating this regularizer during fine-tuning requires a concrete mechanism. We study three instantiations, including (J) an exact evaluation via Jacobian-vector products, (F) a frozen curvature approximation using the Fisher information matrix, and (O) an output-drift formulation that constrains the model's functional deviation on the proxy directly. We prove that all three coincide under the NTK linearization, where existing analyses of such regularizers stop, and go further to characterize standard non-linear fine-tuning, bounding how far each instantiation departs from its linearized reference and obtain a three-way hierarchy, zero for (J), proxy-independent for (F), and proxy-quality-scaling for (O). Empirical validation on the commonly used vision-language model CLIP confirms these theoretical predictions.
BibTeX
@misc{liu2026spectral,
  author = {Shangge Liu and Danyue Qian and Qi Fan and Yinghuan Shi and Wenbin Li and Lei Wang and Yang Gao and Dacheng Tao},
  title = {{A Spectral Framework for Universal Proxy Regularization in Model Merging}},
  year = {2026},
  eprint = {2611.17728},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17728},
}
Figure