Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging

Li Shen, Anke Tang, Enneng Yang, Guibing Guo, Yong Luo, Lefei Zhang, Xiaochun Cao, Bo Du, Dacheng Tao

Introduction

In the rapidly evolving landscape of open-source AI, millions of specialized "expert" models—fine-tuned for specific tasks like medical imaging or traffic sign recognition—are now available. A significant challenge in machine learning is Multi-Task Model Merging: the ability to combine these disparate experts into a single, unified model without requiring access to the original training data.

Traditional merging methods typically seek a static, "one-size-fits-all" weight configuration. However, this often leads to task interference, where averaging the weights of different experts degrades the performance of each. This paper introduces WEMoE (Weight-Ensembling Mixture of Experts) and its efficient variant, E-WEMoE, which move beyond static merging by dynamically adjusting internal weights for every input image.

Selective Upcycling and the Role of MLPs

A core insight of this research is that not all neural network layers contribute equally to task-specific expertise. In Vision Transformers (ViTs), the authors empirically demonstrated that Multi-Layer Perceptron (MLP) layers undergo much more significant changes during fine-tuning than Attention layers.

L2 length of task vectors across layers for MLP and Attention modules, showing that MLPs exhibit significantly higher task-specific variation.

As shown in the figure above, the L2 norm of task vectors (the difference between fine-tuned and pre-trained weights) is consistently higher for MLP modules across almost all layers. This discovery allows for "Selective Upcycling": keeping Attention layers statically merged while transforming MLP layers into a dynamic Mixture-of-Experts structure. This surgical approach minimizes complexity while maximizing the model's ability to adapt to different tasks.

The WEMoE Framework

WEMoE (Weight-Ensembling Mixture of Experts)

WEMoE utilizes a small internal "router" that analyzes each input image to calculate a set of dynamic weights. These weights are used to combine task-specific MLP vectors on the fly, allowing the model to shift its internal logic based on the visual context. To optimize the router without original training data, the authors employ an entropy-minimization technique, ensuring the model's final predictions are as confident as possible.

E-WEMoE (Efficient WEMoE)

While WEMoE is highly effective, storing full MLP weights for every task is memory-intensive. E-WEMoE addresses this through two primary innovations:

  1. Sparsification: The authors found that 90-99% of task-specific weight values are near zero. By pruning these redundant parameters, storage requirements are drastically reduced.
  2. Shared Routing: Instead of a separate router for every transformer layer, a single "Shared Router" is used across the entire model, further cutting down on trainable parameters.

Key Results and Performance

The researchers evaluated their frameworks across eight diverse image classification benchmarks using CLIP-ViT architectures. The results demonstrate that dynamic merging consistently outperforms static methods like Task Arithmetic and AdaMerging.

Average accuracy vs. normalized number of parameters for different CLIP-ViT architectures. E-WEMoE variants maintain high performance while significantly reducing parameter overhead.

As illustrated in the performance plots, WEMoE achieves an average accuracy (89.4% on ViT-B/32) that nearly matches traditional multi-task learning (90.5%), which requires training on all data simultaneously. Notably, E-WEMoE-90% maintains this high performance while utilizing four times fewer total parameters and up to 24 times fewer trainable parameters for the router compared to the full WEMoE model.

Conclusion

By transitioning from static to instance-specific dynamic merging, WEMoE and E-WEMoE effectively close the performance gap in multi-task model merging. The success of sparsification—maintaining performance even after pruning 99% of task vectors—highlights the inherent redundancy in specialized models and paves the way for more efficient large-scale model integration. Future work may extend these "upcycling" principles to Large Language Models (LLMs) and explore further parameter-efficient fine-tuning (PEFT) techniques.

Summary updated

BibTeX
@article{shen2025efficient,
  author = {Li Shen and Anke Tang and Enneng Yang and Guibing Guo and Yong Luo and Lefei Zhang and Xiaochun Cao and Bo Du and Dacheng Tao},
  title = {{Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2025},
  doi = {10.1109/tpami.2025.3629605},
}
Figure