SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation

Aecheon Jung, Seunghwan Lee, Dongyoon Han, Sungeun Hong

TL;DR

SyMerge is a novel, lightweight test-time adaptation framework that shifts the paradigm of model merging from merely avoiding task interference to actively fostering positive task synergy. By jointly optimizing merging coefficients and just a single task-specific layer using an expert-guided self-labeling objective, SyMerge achieves state-of-the-art multi-task performance across vision, dense prediction, and NLP benchmarks. Remarkably, it even succeeds in merging models trained from completely different initializations—a scenario where traditional merging methods catastrophically fail.

What Problem Does This Paper Address?

Training a single multi-task model from scratch is computationally expensive and requires delicate loss-balancing acts. Model merging has emerged as a popular alternative: it takes independently fine-tuned, task-specific models (experts) and combines their weights into a single multi-task network without retraining the entire model.

However, existing merging techniques suffer from two major flaws:

  1. The "Non-Interference" Ceiling: Most methods focus entirely on minimizing task interference (preventing tasks from harming one another) or weight disentanglement. The authors argue this sets a performance ceiling. Instead, merging should aim for synergy, where tasks actively help each other improve.
  2. Brittleness Under Distribution Shifts: Training-free merging methods perform poorly when encountering out-of-distribution or corrupted data at test time. While existing test-time adaptation (TTA) methods try to fix this using entropy minimization, this proxy objective is notoriously unstable and often diverges from actual task performance over time.
Figure 1. Training-Free methods collapse under corruption. Worse than test-time methods on clean data as well, and far more degraded under corruption.

Methods

SyMerge introduces a minimalist, test-time adaptive framework that achieves synergy through two main mechanisms:

  • Joint Co-Adaptation: Instead of only tuning the merging coefficients of the shared encoder, SyMerge jointly optimizes the encoder's merging coefficients and a single task-specific layer (such as the final classifier or an intermediate transformer block). This minimal adaptation realigns representations and unlocks cross-task compatibility.
  • Expert-Guided Self-Labeling: To bypass the instability of entropy minimization, SyMerge uses the original, unmerged fine-tuned models as "expert teachers." During test-time adaptation on unlabeled data, these experts generate pseudo-labels. The merged model is then optimized using a standard cross-entropy loss (or $L_1$ loss for regression tasks) to match the experts' confident predictions.

Additionally, SyMerge implements a sequential per-task update strategy (updating parameters immediately after each task's forward pass) which mitigates catastrophic forgetting, and a relative confidence-based filtering mechanism to ensure the model only learns from highly reliable expert predictions.

Datasets

The authors evaluated SyMerge across three distinct domains:

  • Vision (Image Classification): 8, 14, and 20 task configurations based on the CLIP (ViT-B/32 and ViT-L/14) architecture, including datasets like SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD.
  • Dense Prediction: The NYUv2 dataset, which requires simultaneous handling of three highly heterogeneous tasks: 13-class semantic segmentation, depth estimation, and surface normal estimation using a ResNet-50 backbone.
  • Natural Language Processing (NLP): 8 tasks from the GLUE benchmark evaluated on RoBERTa-base and scaled up to the generative FLAN-T5 Base model.

Key Insights and Ideas

  • Cross-Task Performance Predicts Merge Quality: In a pilot study, the authors discovered that a model's ability to generalize across tasks (e.g., pairing an encoder trained on Task A with a classifier trained on Task B) strongly correlates ($r = 0.863$) with its ultimate merging success.
Figure 2. Cross-task vs. Merge Performance. Positive correla- tion observed across 20 vision tasks with regression fit and 95% confidence interval.
  • Single-Layer Adaptation is Sufficient: You don't need to fine-tune the entire network to achieve synergy. Adapting just one layer (even an internal transformer block) provides enough flexibility to realign representations across tasks while preserving task-agnostic knowledge.
  • Stable Supervision via Self-Labeling: As adaptation progresses, entropy minimization losses tend to drift away from the true supervised objective. The expert-guided cross-entropy loss used in SyMerge maintains a consistently high correlation with the ground-truth loss throughout training.
Figure 4. Correlation of proxy losses with the ground truth loss. The ground truth loss is defined as the supervised cross-entropy computed with ground truth labels. We compare coefficients for Entropy and Ours using merged weights before and after training. A coefficient closer to +1 indicates a more reliable proxy.
  • Bridging Disjoint Basins: Traditional weight arithmetic assumes models share a common pre-trained initialization. SyMerge breaks this barrier. By co-adapting the internal layer and coefficients, it can successfully merge models initialized from completely different pre-trained weights (e.g., merging an OpenAI-ViT fine-tuned model with a DataComp-ViT fine-tuned model).

Key Results

  • State-of-the-Art Performance: SyMerge consistently outperforms competing training-free and test-time adaptive merging methods. For instance, on the 20-task vision benchmark (ViT-B/32), SyMerge achieves 88.6% accuracy, approaching the individual expert upper bound of 90.4%, while standard Task Arithmetic drops to 60.8%.
  • Dense Prediction Success: On the challenging NYUv2 dense prediction benchmark, SyMerge achieves a segmentation mIoU of 49.8% (compared to individual models at 52.0%), significantly beating other advanced merging methods like ProbSurgery (43.6%) and EMR-Merging (41.5%).
  • Disjoint Basin Breakthrough: When merging models from different initializations, standard weight averaging collapses to an unusable 6.9% average accuracy. SyMerge successfully recovers performance to 79.1%.
  • High Efficiency: SyMerge is highly parameter-efficient, requiring only 0.39M trainable parameters for the 8-task vision setup, maintaining a low memory footprint (7.12 GB) and fast runtime (111 ms per step when caching expert predictions).

Limitations and Future Work

Despite its strong performance, SyMerge has a few limitations:

  • Dependency on Expert Quality: Because it relies on the unmerged models for pseudo-labels, its performance is inherently bounded by the quality and potential biases of the individual expert models.
  • Requirement for Test Data: SyMerge requires access to unlabeled target inputs at test time, which might pose challenges in strict privacy-preserving or ultra-low-latency deployment scenarios.
  • Theoretical Open Questions: Fully characterizing the exact mathematical interaction between coefficient tuning and single-layer adaptation remains an open direction for future research.

Summary updated

Abstract

Model merging combines independently trained models into a single multi-task model. However, most existing approaches focus primarily on avoiding task interference. We argue that its greater potential lies in enabling task synergy, where tasks actively improve one another. We identify cross-task performance, defined by compatibility between encoders and predictors across tasks, as a key indicator of merge quality. We demonstrate that adapting only a single task-specific layer is sufficient to induce such synergy. This study proposes SyMerge, a lightweight framework that jointly optimizes merging coefficients and a single task-specific layer. We adopt an expert-guided self-labeling objective, providing stable supervision beyond entropy minimization. Intriguingly, we further show that SyMerge successfully merges models trained from different initializations, a regime where standard methods break down. Our minimalist yet principled method achieves state-of-the-art results across vision, dense prediction, and NLP benchmarks. Our code is available at https://aim-skku.github.io/SyMerge
BibTeX
@inproceedings{jung2026symerge,
  author = {Aecheon Jung and Seunghwan Lee and Dongyoon Han and Sungeun Hong},
  title = {{SyMerge: From Non-Interference to Synergistic Merging via Single-Layer Adaptation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SroXgWAQll},
}
Figure