A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning

Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, Alessandro Sordoni

TL;DR

This paper provides the first comprehensive survey of Model MoErging, an emerging machine learning paradigm that recycles independently fine-tuned "expert" models into a dynamic, collaborative system using adaptive routing. The authors establish a unified taxonomy to categorize diverse MoErging methods, map out the current software ecosystem, and highlight critical open challenges for future research.

What Problem Does This Paper Address?

Training massive, general-purpose AI models from scratch is incredibly compute-intensive and out of reach for most researchers. As a workaround, the community has turned to fine-tuning smaller, specialized "expert" models (often using parameter-efficient fine-tuning techniques like LoRA) for specific tasks or domains.

However, two major limitations persist:

  1. Standard Mixture-of-Experts (MoE) models require all experts to be trained jointly from scratch under a centralized framework.
  2. Model Merging techniques typically combine models into a single, static weight configuration that cannot adapt dynamically to different inputs at runtime.

Model MoErging (a portmanteau of MoE and Merging) solves this by dynamically routing queries among a decentralized pool of pre-existing, independently trained experts. Because this subfield has exploded recently with fragmented terminology and isolated experimental setups, this survey addresses the lack of a unified framework to compare and contrast these methods.

Methods

As a survey paper, the primary contribution is a novel, three-tiered taxonomy that categorizes MoErging design choices:

  1. Expert Design: Classifies whether experts are trained using standard fine-tuning or custom procedures, and whether their training data must be shared or can remain private.
  2. Routing Strategy: Analyzes how the system decides which expert to use based on:
  3. Routing Dataset: The data used to train the router (None, Target-task, Expert, or General data).
  4. Input Granularity: Whether routing decisions happen per task, per example, or per token/step.
  5. Depth Granularity: Whether a single routing decision applies to the entire model or independent decisions are made at each module/layer.
  6. Expert Selection & Aggregation: Whether the selection is sparse (top-$k$) or dense (all experts), and if they are combined via output ensembling or parameter merging.
  7. Application Design: Evaluates the system based on its generalization goals (In-Distribution vs. Out-of-Distribution) and user data availability (Zero-Shot, Few-Shot, or Full dataset).

The authors group existing literature into four high-level routing paradigms: Embedding-based, Classifier-based, Task-specific, and Router-free approaches.

The proposed taxonomy tree for cataloging model MoErging design choices.

Datasets

As a survey, this paper does not introduce a new dataset but reviews literature evaluated on a wide variety of standard benchmarks, including:

  • General LLM Evaluation: MMLU, HELM, BIG-Bench Hard, Flan collection.
  • Chat & Alignment: MT-Bench, FLASK.
  • Reasoning & Math: GSM8K, MGSM, HumanEval.

Key Insights and Ideas

  • Decentralization is Key: Unlike traditional MoE, MoErging allows a distributed community to contribute specialized adapters asynchronously without sharing their private training data or compute resources.
  • The Granularity Trade-off: Finer routing granularity (e.g., token-level or module-level) allows for highly nuanced input adaptation but introduces significant computational overhead and risks propagating early routing errors to later layers.
Different levels of input granularity for routing decisions.
  • Output Ensembling vs. Parameter Merging: The survey notes that ensembling expert outputs dynamically at runtime generally yields superior performance compared to merging expert parameters, though parameter aggregation results in a much more compact and efficient model.
  • Data Privacy Bottlenecks: Many high-performing MoErging methods assume access to the experts' original training data. However, because public model hubs (like Hugging Face) rarely include training datasets alongside model weights, "Private Data" and "Router-Free" methods are much more practical for real-world deployment.

Key Results

The survey synthesizes findings from nearly 30 recent MoErging methods (such as AdapterFusion, LoraHub, PHATGOOSE, and RouteLLM). Quantitatively, the reviewed literature demonstrates that MoErging consistently outperforms static model merging and standard multitask training baselines. Crucially, it enables robust zero-shot and out-of-distribution (OOD) generalization while significantly lowering inference costs—for instance, by using a classifier-based router to dynamically choose between a cheap model (like Mixtral) and an expensive one (like GPT-4) based on query complexity.

Limitations and Future Work

The authors identify several open challenges blocking widespread adoption:

  • Lack of User-Friendly Tooling: While backend libraries like Arcee's MergeKit and Predibase's Lorax support basic merging, many advanced MoErging routing algorithms lack accessible, production-ready implementations.
  • Unrealistic Data Assumptions: A gap exists between academic methods requiring shared data/custom training and the actual practice of sharing isolated, off-the-shelf model weights.
  • Ecosystem Management: As expert pools grow, future research must determine how to identify and remove redundant models, handle malicious/poisoned expert contributions, and establish robust theoretical frameworks to guarantee system stability.

Summary updated

Abstract

The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to a particular domain or task. Model MoErging methods aim to recycle expert models to create an aggregate system with improved performance or generalization. A key component of MoErging methods is the creation of a router that decides which expert model(s) to use for a particular input or application. The promise, effectiveness, and large design space of MoErging has spurred the development of many new methods over the past few years. This rapid pace of development has made it challenging to compare different MoErging methods, which are rarely compared to one another and are often validated in different experimental setups. To remedy such gaps, we present a comprehensive survey of MoErging methods that includes a novel taxonomy for cataloging key design choices and clarifying suitable applications for each method. Apart from surveying MoErging research, we inventory software tools and applications that make use of MoErging. We additionally discuss related fields of study such as model merging, multitask learning, and mixture-of-experts models. Taken as a whole, our survey provides a unified overview of existing MoErging methods and creates a solid foundation for future work in this burgeoning field.
BibTeX
@article{yadav2025survey,
  author = {Prateek Yadav and Colin Raffel and Mohammed Muqeeth and Lucas Caccia and Haokun Liu and Tianlong Chen and Mohit Bansal and Leshem Choshen and Alessandro Sordoni},
  title = {{A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learning}},
  journal = {Trans. Mach. Learn. Res.},
  year = {2025},
  url = {https://openreview.net/forum?id=aavwVaWgHP},
}
Figure