TL;DR
SyMerge is a novel, lightweight test-time adaptation framework that shifts the paradigm of model merging from merely avoiding task interference to actively fostering positive task synergy. By jointly optimizing merging coefficients and just a single task-specific layer using an expert-guided self-labeling objective, SyMerge achieves state-of-the-art multi-task performance across vision, dense prediction, and NLP benchmarks. Remarkably, it even succeeds in merging models trained from completely different initializations—a scenario where traditional merging methods catastrophically fail.
What Problem Does This Paper Address?
Training a single multi-task model from scratch is computationally expensive and requires delicate loss-balancing acts. Model merging has emerged as a popular alternative: it takes independently fine-tuned, task-specific models (experts) and combines their weights into a single multi-task network without retraining the entire model.
However, existing merging techniques suffer from two major flaws:
- The "Non-Interference" Ceiling: Most methods focus entirely on minimizing task interference (preventing tasks from harming one another) or weight disentanglement. The authors argue this sets a performance ceiling. Instead, merging should aim for synergy, where tasks actively help each other improve.
- Brittleness Under Distribution Shifts: Training-free merging methods perform poorly when encountering out-of-distribution or corrupted data at test time. While existing test-time adaptation (TTA) methods try to fix this using entropy minimization, this proxy objective is notoriously unstable and often diverges from actual task performance over time.
Methods
SyMerge introduces a minimalist, test-time adaptive framework that achieves synergy through two main mechanisms:
- Joint Co-Adaptation: Instead of only tuning the merging coefficients of the shared encoder, SyMerge jointly optimizes the encoder's merging coefficients and a single task-specific layer (such as the final classifier or an intermediate transformer block). This minimal adaptation realigns representations and unlocks cross-task compatibility.
- Expert-Guided Self-Labeling: To bypass the instability of entropy minimization, SyMerge uses the original, unmerged fine-tuned models as "expert teachers." During test-time adaptation on unlabeled data, these experts generate pseudo-labels. The merged model is then optimized using a standard cross-entropy loss (or $L_1$ loss for regression tasks) to match the experts' confident predictions.
Additionally, SyMerge implements a sequential per-task update strategy (updating parameters immediately after each task's forward pass) which mitigates catastrophic forgetting, and a relative confidence-based filtering mechanism to ensure the model only learns from highly reliable expert predictions.
Datasets
The authors evaluated SyMerge across three distinct domains:
- Vision (Image Classification): 8, 14, and 20 task configurations based on the CLIP (ViT-B/32 and ViT-L/14) architecture, including datasets like SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD.
- Dense Prediction: The NYUv2 dataset, which requires simultaneous handling of three highly heterogeneous tasks: 13-class semantic segmentation, depth estimation, and surface normal estimation using a ResNet-50 backbone.
- Natural Language Processing (NLP): 8 tasks from the GLUE benchmark evaluated on RoBERTa-base and scaled up to the generative FLAN-T5 Base model.
Key Insights and Ideas
- Cross-Task Performance Predicts Merge Quality: In a pilot study, the authors discovered that a model's ability to generalize across tasks (e.g., pairing an encoder trained on Task A with a classifier trained on Task B) strongly correlates ($r = 0.863$) with its ultimate merging success.
- Single-Layer Adaptation is Sufficient: You don't need to fine-tune the entire network to achieve synergy. Adapting just one layer (even an internal transformer block) provides enough flexibility to realign representations across tasks while preserving task-agnostic knowledge.
- Stable Supervision via Self-Labeling: As adaptation progresses, entropy minimization losses tend to drift away from the true supervised objective. The expert-guided cross-entropy loss used in SyMerge maintains a consistently high correlation with the ground-truth loss throughout training.
- Bridging Disjoint Basins: Traditional weight arithmetic assumes models share a common pre-trained initialization. SyMerge breaks this barrier. By co-adapting the internal layer and coefficients, it can successfully merge models initialized from completely different pre-trained weights (e.g., merging an OpenAI-ViT fine-tuned model with a DataComp-ViT fine-tuned model).
Key Results
- State-of-the-Art Performance: SyMerge consistently outperforms competing training-free and test-time adaptive merging methods. For instance, on the 20-task vision benchmark (ViT-B/32), SyMerge achieves 88.6% accuracy, approaching the individual expert upper bound of 90.4%, while standard Task Arithmetic drops to 60.8%.
- Dense Prediction Success: On the challenging NYUv2 dense prediction benchmark, SyMerge achieves a segmentation mIoU of 49.8% (compared to individual models at 52.0%), significantly beating other advanced merging methods like ProbSurgery (43.6%) and EMR-Merging (41.5%).
- Disjoint Basin Breakthrough: When merging models from different initializations, standard weight averaging collapses to an unusable 6.9% average accuracy. SyMerge successfully recovers performance to 79.1%.
- High Efficiency: SyMerge is highly parameter-efficient, requiring only 0.39M trainable parameters for the 8-task vision setup, maintaining a low memory footprint (7.12 GB) and fast runtime (111 ms per step when caching expert predictions).
Limitations and Future Work
Despite its strong performance, SyMerge has a few limitations:
- Dependency on Expert Quality: Because it relies on the unmerged models for pseudo-labels, its performance is inherently bounded by the quality and potential biases of the individual expert models.
- Requirement for Test Data: SyMerge requires access to unlabeled target inputs at test time, which might pose challenges in strict privacy-preserving or ultra-low-latency deployment scenarios.
- Theoretical Open Questions: Fully characterizing the exact mathematical interaction between coefficient tuning and single-layer adaptation remains an open direction for future research.
Summary updated