Abstract
Traditional model merging methods for multi-task learning (MTL) address task conflicts with straightforward strategies such as weight averaging, sign consensus, or minimal test-time adjustments. This presumably counts on the assumption that a merged encoder still retains abundant task knowledge from individual encoders, implying that its shared representation is sufficiently general across tasks. However, our insight is that adding just a single trainable task-specific layer further can bring striking performance gains, as demonstrated by our pilot study. Motivated by this finding, we propose Model Tinting (SyMerge), a new test-time approach that introduces a single task-specific layer for each task as trainable adjustments. Our method jointly trains merging coefficients and task-specific layers, which effectively reduces task conflicts with minimal additional costs. Additionally, we propose a sampling method that utilizes the difference in confidence levels of both merged and individual encoders. Extensive experiments demonstrate our method's effectiveness, which achieves state-of-the-art performance across both computer vision and natural language processing tasks and significantly surpasses prior works.
Introduction: Beyond Non-Interference
Model merging has become a popular alternative to traditional Multi-Task Learning (MTL), allowing independently fine-tuned "expert" models to be combined into a single architecture. The primary obstacle in this field is interference: parameter conflicts that often lead to a "jack of all trades, master of none" scenario. While most research focuses on mitigation—using masks or pruning to achieve non-interference—SyMerge (Model Tinting) shifts the goal toward synergy, where tasks actively enhance each other's performance.
Methodology
SyMerge is a lightweight, test-time adaptive framework that adapts to unlabeled data at inference time. It relies on three key pillars:
- Single-Layer Adaptation (Model Tinting): Based on findings that cross-task compatibility is highly sensitive to final layers, SyMerge updates only a single task-specific layer (usually the classifier head) alongside merging coefficients ($\lambda$). This "tinting" process keeps the adaptation lightweight and preserves general knowledge in the pre-trained encoder.
- Joint Optimization: The framework simultaneously learns optimal weights for combining encoders and the parameters for the task-specific heads.
- Expert-Guided Self-Labeling: To avoid the instability of standard entropy minimization, SyMerge uses the original, unmerged "expert" models as teachers. The merged model is trained to match the confident predictions of these experts, providing a stable and reliable training signal.
Key Insights: The Cross-Task Predictor
A central discovery of this research is that "cross-task" performance—pairing Encoder A with Classifier B—is a near-perfect predictor of how well those models will merge.
As shown in the figure above, models that work well together in a "franken-model" state merge more successfully. Mathematically, the authors prove that if tasks are synergistic (improving each other's loss), the merged model can achieve a strictly lower loss than a model that merely achieves non-interference.
Key Results
SyMerge consistently outperforms both training-free and test-time adaptive baselines across vision, NLP, and dense prediction benchmarks.
- Vision Scaling: In a 20-task vision benchmark, SyMerge achieved 88.6% accuracy, significantly outperforming EMR-Merging (86.6%) and standard Task Arithmetic (60.8%).
- NLP: On the GLUE benchmark, it reached an average score of 83.9, nearly matching individually fine-tuned models (85.6).
- Robustness: SyMerge demonstrates remarkable stability under data corruption. While training-free methods often "collapse" under distribution shifts, SyMerge maintains high performance.
Conclusion
SyMerge demonstrates that moving from "not breaking" to "making better together" is possible through targeted, task-specific adjustments. By introducing a single "tinted" layer and leveraging expert guidance, the framework achieves state-of-the-art performance with minimal test-time overhead.
Code available at: https://github.com/AIM-SKKU/ModelTinting
Summary updated