SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation

Aecheon Jung, Seunghwan Lee, Dongyoon Han, Sungeun Hong

Abstract

Model merging offers an efficient alternative to multi-task learning by combining independently fine-tuned models, but most prior approaches focus mainly on avoiding task interference. We argue instead that the real potential of merging lies in achieving synergy, where tasks enhance one another. Our intuition comes from a pilot study showing that when a classifier trained on one task is paired with the encoder of another, the resulting cross-task performance strongly predicts merge quality. Moreover, adapting even a single task-specific layer can substantially improve this compatibility, suggesting a simple yet powerful lever for synergy. Building on this insight, we introduce SyMerge, a lightweight framework that jointly optimizes one task-specific layer and merging coefficients. To ensure stability without labels, SyMerge employs a robust self-labeling strategy guided by expert model predictions, avoiding the pitfalls of entropy-based adaptation. This minimalist yet principled design achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks, while also producing adapted layers that transfer effectively to other merging methods.

Introduction

Model merging has emerged as a computationally efficient alternative to traditional Multi-Task Learning (MTL). Rather than training a single massive model on all tasks simultaneously—a process that is often prohibitively expensive—researchers can take independently fine-tuned models and combine their "task vectors" (the difference between fine-tuned and pre-trained weights).

However, the primary obstacle to successful merging is interference: the weight updates for Task A often degrade the performance of Task B. While existing literature focuses on "non-interference"—attempting to combine models without breaking their individual functions—SyMerge shifts the paradigm toward synergy. The goal is to foster an environment where knowledge from disparate tasks enhances one another, potentially allowing a merged model to outperform its individual components.

The Quest for Synergy: Functional Alignment

SyMerge is built on the principle that "functional alignment" between tasks is the key to synergy. The authors conducted a pilot study to investigate what makes models mergeable. They discovered a critical metric: "cross-task performance." By taking the encoder from one task and pairing it with the classifier head of another, they found that the resulting performance is a remarkably strong predictor of how well those two models will merge.

Correlation between cross-task performance and merge quality

This finding suggests that synergy is not just about weight overlap, but about how well the internal representations of different tasks can be reconciled. By improving this cross-task compatibility, we can unlock higher theoretical upper bounds for merging performance.

The SyMerge Framework

SyMerge is a lightweight, test-time adaptive framework that employs three core strategies to achieve synergy:

  1. Joint Optimization: Unlike previous methods that only tune merging coefficients (the scalar weights assigned to each task vector), SyMerge simultaneously optimizes these coefficients and a single task-specific layer—typically the final classifier or "head."
  2. Expert-Guided Self-Labeling: Test-time adaptation often lacks ground-truth labels. While many methods rely on entropy minimization (forcing the model to be more confident), this can lead to unstable convergence on incorrect predictions. SyMerge instead uses the original, independently fine-tuned models as "expert teachers." The merged model is trained to match the high-confidence predictions of these experts, providing a stable "anchor" for adaptation.
  3. Minimalist Adaptation: The authors demonstrate that adapting just one layer is a powerful lever. It provides sufficient flexibility to align features across tasks without the risk of catastrophic forgetting or disrupting the general knowledge embedded in the pre-trained backbone.

Experimental Results

The authors evaluated SyMerge across vision (CLIP ViT backbones), dense prediction (NYUv2), and NLP (RoBERTa on GLUE) benchmarks.

In vision tasks, SyMerge consistently outperformed both training-free and test-time adaptive baselines. On a 20-task benchmark using ViT-B/32, SyMerge achieved 88.6% accuracy, a massive leap over the standard Task Arithmetic baseline (60.8%) and the previous state-of-the-art, AdaMerging (69.6%).

Performance comparison across various test-time adaptation methods on original and corrupted data

As shown in the results, SyMerge is particularly robust under distribution shifts (corruption), maintaining high accuracy where other methods falter. Furthermore, the task-specific layers adapted by SyMerge are "plug-and-play"; they can be transferred to other merged encoders (like those created by Task Arithmetic) and still provide significant performance gains, indicating they have learned a generalized functional alignment.

Conclusion

SyMerge demonstrates that model merging can move beyond simple interference avoidance toward active synergy. By focusing on functional alignment through single-layer adaptation and expert-guided self-labeling, the framework achieves state-of-the-art performance with minimal overhead. While the method currently depends on the quality of initial "expert" models and requires a test-time adaptation phase, it opens new doors for creating highly capable multi-task models from specialized components.

Summary updated

Abstract

Model merging offers an efficient alternative to multi-task learning by combining independently fine-tuned models, but most prior approaches focus mainly on avoiding task interference. We argue instead that the real potential of merging lies in achieving synergy, where tasks enhance one another. Our intuition comes from a pilot study showing that when a classifier trained on one task is paired with the encoder of another, the resulting cross-task performance strongly predicts merge quality. Moreover, adapting even a single task-specific layer can substantially improve this compatibility, suggesting a simple yet powerful lever for synergy. Building on this insight, we introduce SyMerge, a lightweight framework that jointly optimizes one task-specific layer and merging coefficients. To ensure stability without labels, SyMerge employs a robust self-labeling strategy guided by expert model predictions, avoiding the pitfalls of entropy-based adaptation. This minimalist yet principled design achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks, while also producing adapted layers that transfer effectively to other merging methods. Our code is available at https://aim-skku.github.io/SyMerge/
BibTeX
@inproceedings{jung2026symerge,
  author = {Aecheon Jung and Seunghwan Lee and Dongyoon Han and Sungeun Hong},
  title = {{SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SroXgWAQll},
}
Figure