TL;DR
This paper introduces a Spectral Framework to explain how task-agnostic "proxy" data (like ImageNet) can prevent parameter interference during model merging. By regularizing fine-tuning with a universal proxy, researchers can create "merge-friendly" models that achieve near-oracle performance without ever needing access to the private data of other tasks.
What Problem Does This Paper Address?
Model merging—specifically Task Arithmetic—is a powerful way to combine independently fine-tuned "expert" models into a single multi-task model without expensive joint training. However, it often fails due to parameter interference: the weight updates that make a model good at Task A might inadvertently degrade its performance on Task B.
To fix this, existing "pre-merging" methods constrain fine-tuning using data from the other tasks you plan to merge with. This creates a catch-22: the whole point of model merging is to keep tasks independent and private, but these fixes require sharing data between them.
The authors ask: Can we replace specific peer-task data with a universal, task-agnostic proxy distribution? And more importantly, why and when does such a proxy work?
Methods
The authors propose a Spectral Framework for Universal Proxy Regularization. They characterize the relationship between a proxy distribution ($Q$) and a target task ($P_t$) using two key spectral parameters:
- Spectral Coverage ($\alpha$): Measures how well the proxy's gradient space spans the target task's space.
- Spectral Leakage ($\beta$): Quantifies the gradient energy in directions the proxy fails to cover.
They derive a theoretical bound proving that smaller $(\alpha, \beta)$ values lead to lower parameter interference. To put this into practice, they augment the fine-tuning objective with a proxy regularizer ($R_Q$) and study three ways to implement it:
- (J) Jacobian: Exact evaluation using Jacobian-vector products (accurate but expensive).
- (F) Frozen Fisher: Uses a pre-computed Fisher Information Matrix snapshot (efficient but can drift).
- (O) Output-drift: Directly penalizes changes in the model's functional output on proxy samples (efficient and robust).
Crucially, they analyze these methods under non-linear fine-tuning, moving beyond the simplified linear assumptions (NTK) common in previous literature.
Datasets
The framework was validated across both vision and language modalities:
- Vision: CLIP models (ViT-B/32, B/16, L/14) evaluated on 8 benchmarks including Cars, DTD, EuroSAT, and MNIST.
- Language: Llama-3.2 (1B, 3B) on the TRACE benchmark and Llama-3 (8B) on GLUE tasks using LoRA.
- Proxies: Common general-purpose datasets like ImageNet (for vision) and WikiText-103 (for language) served as universal proxies.
Key Insights and Ideas
- Broad Coverage Trumps Semantic Similarity: A proxy doesn't need to "look like" the target task. ImageNet works as a proxy for medical imagery or satellite data because it provides broad spectral coverage of the model's parameter space, not because the classes are similar.
- The Three-Way Hierarchy: In real-world non-linear training, the three implementation methods behave differently. The Jacobian (J) method is the gold standard for accuracy, while the Output-drift (O) method is a "best of both worlds" choice—it's computationally cheap (just a forward pass) but its accuracy scales with the quality of the proxy.
- The Natural Image Manifold: Even synthetic images (from Stable Diffusion) work significantly better than random noise as proxies. This is because they activate the "natural image manifold" that the pre-trained model was originally built on.
Key Results
- Performance Boost: On ViT-B/32, adding proxy regularization to standard fine-tuning lifted merged absolute accuracy by 6.7–13.4 percentage points.
- Closing the Oracle Gap: The method consistently approached the performance of "oracle" methods that have illegal access to peer-task data. For example, on ViT-L/14, the proxy-based method reached 91.64% accuracy, nearly matching the oracle's 90.80%.
- LLM Success: Using WikiText-103 as a proxy for Llama-3.2 models improved multi-task merging scores by up to 2 points without ever seeing the specific downstream task data during the regularization phase.
- Efficiency: The Output-drift (O) instantiation was found to be the most favorable for high-quality proxies, being ~43% faster and using significantly less memory than Jacobian-based methods.
Limitations and Future Work
- Proxy Construction: While existing datasets like ImageNet work well, the authors suggest future work could involve actively constructing optimal proxies that minimize the $(\alpha, \beta)$ criteria for specific architectures.
- Beyond Additive Merging: The current framework is tailored for Task Arithmetic (additive merging). Extending these spectral insights to other merging paradigms, like SVD-based merging or model breadcrumbing, remains an open challenge.
- Cross-Modal Merging: The paper treats vision and language separately; exploring universal proxies for cross-modal model composition is a promising next step.
Summary updated