Tint Your Models Task-wise for Improved Multi-task Model Merging

Aecheon Jung, Seunghwan Lee, Dongyoon Han, Sungeun Hong

Abstract

Traditional model merging methods for multi-task learning (MTL) address task conflicts with straightforward strategies such as weight averaging, sign consensus, or minimal test-time adjustments. This presumably counts on the assumption that a merged encoder still retains abundant task knowledge from individual encoders, implying that its shared representation is sufficiently general across tasks. However, our insight is that adding just a single trainable task-specific layer further can bring striking performance gains, as demonstrated by our pilot study. Motivated by this finding, we propose Model Tinting (SyMerge), a new test-time approach that introduces a single task-specific layer for each task as trainable adjustments. Our method jointly trains merging coefficients and task-specific layers, which effectively reduces task conflicts with minimal additional costs. Additionally, we propose a sampling method that utilizes the difference in confidence levels of both merged and individual encoders. Extensive experiments demonstrate our method's effectiveness, which achieves state-of-the-art performance across both computer vision and natural language processing tasks and significantly surpasses prior works.

Introduction: Beyond Non-Interference

Model merging has become a popular alternative to traditional Multi-Task Learning (MTL), allowing independently fine-tuned "expert" models to be combined into a single architecture. The primary obstacle in this field is interference: parameter conflicts that often lead to a "jack of all trades, master of none" scenario. While most research focuses on mitigation—using masks or pruning to achieve non-interference—SyMerge (Model Tinting) shifts the goal toward synergy, where tasks actively enhance each other's performance.

Methodology

SyMerge is a lightweight, test-time adaptive framework that adapts to unlabeled data at inference time. It relies on three key pillars:

  1. Single-Layer Adaptation (Model Tinting): Based on findings that cross-task compatibility is highly sensitive to final layers, SyMerge updates only a single task-specific layer (usually the classifier head) alongside merging coefficients ($\lambda$). This "tinting" process keeps the adaptation lightweight and preserves general knowledge in the pre-trained encoder.
  2. Joint Optimization: The framework simultaneously learns optimal weights for combining encoders and the parameters for the task-specific heads.
  3. Expert-Guided Self-Labeling: To avoid the instability of standard entropy minimization, SyMerge uses the original, unmerged "expert" models as teachers. The merged model is trained to match the confident predictions of these experts, providing a stable and reliable training signal.

Key Insights: The Cross-Task Predictor

A central discovery of this research is that "cross-task" performance—pairing Encoder A with Classifier B—is a near-perfect predictor of how well those models will merge.

Correlation between cross-task performance and merge performance ($r=0.863$). High cross-task compatibility indicates a high likelihood of successful merging.

As shown in the figure above, models that work well together in a "franken-model" state merge more successfully. Mathematically, the authors prove that if tasks are synergistic (improving each other's loss), the merged model can achieve a strictly lower loss than a model that merely achieves non-interference.

Key Results

SyMerge consistently outperforms both training-free and test-time adaptive baselines across vision, NLP, and dense prediction benchmarks.

  • Vision Scaling: In a 20-task vision benchmark, SyMerge achieved 88.6% accuracy, significantly outperforming EMR-Merging (86.6%) and standard Task Arithmetic (60.8%).
  • NLP: On the GLUE benchmark, it reached an average score of 83.9, nearly matching individually fine-tuned models (85.6).
  • Robustness: SyMerge demonstrates remarkable stability under data corruption. While training-free methods often "collapse" under distribution shifts, SyMerge maintains high performance.
Accuracy comparison on original vs. corrupted data. SyMerge (Ours) exhibits superior robustness compared to existing test-time adaptation and merging methods.

Conclusion

SyMerge demonstrates that moving from "not breaking" to "making better together" is possible through targeted, task-specific adjustments. By introducing a single "tinted" layer and leveraging expert guidance, the framework achieves state-of-the-art performance with minimal test-time overhead.

Code available at: https://github.com/AIM-SKKU/ModelTinting

Summary updated

Abstract

Traditional model merging methods for multi-task learning (MTL) address task conflicts with straightforward strategies such as weight averaging, sign consensus, or minimal test-time adjustments. This presumably counts on the assumption that a merged encoder still retains abundant task knowledge from individual encoders, implying that its shared representation is sufficiently general across tasks. However, our insight is that adding just a single trainable task-specific layer further can bring striking performance gains, as demonstrated by our pilot study. Motivated by this finding, we propose Model Tinting, a new test-time approach that introduces a single task-specific layer for each task as trainable adjustments. Our method jointly trains merging coefficients and task-specific layers, which effectively reduces task conflicts with minimal additional costs. Additionally, we propose a sampling method that utilizes the difference in confidence levels of both merged and individual encoders. Extensive experiments demonstrate our method's effectiveness, which achieves state-of-the-art performance across both computer vision and natural language processing tasks and significantly surpasses prior works. Our code is available at https://github.com/AIM-SKKU/ModelTinting.
BibTeX
@inproceedings{jung2026tint,
  author = {Aecheon Jung and Seunghwan Lee and Dongyoon Han and Sungeun Hong},
  title = {{Tint Your Models Task-wise for Improved Multi-task Model Merging}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SroXgWAQll},
}
Figure