Abstract
A sparse Mixture-of-Experts (MoE) architecture has emerged as a highly scalable solution by conditionally activating sub-modules without a proportional increase in computational costs. However, improving expert specialization to enhance performance and generalization remains a challenge for MoE, especially in instruction tuning scenarios characterized by significant input heterogeneity. In this work, we propose the Mixture-of-Clustered-Experts (MoCE) to address this limitation through a dual-stage routing mechanism. The first stage in the mechanism performs expert group routing based on sequence-level features, while the second stage activates the top-$k$ experts within the group at the token level. This approach enables the effective partitioning of heterogeneous inputs based on their knowledge requirements, encouraging expert group specialization while maintaining the advantages of token-level routing. We evaluate MoCE across a comprehensive set of benchmarks, demonstrating its consistent superiority over strong baselines and its enhanced generalization capabilities. Detailed analysis further highlights the robustness and effectiveness of MoCE.
Introduction
Instruction tuning requires Large Language Models (LLMs) to process a vast and heterogeneous array of tasks, ranging from creative writing to complex mathematical reasoning. While Mixture-of-Experts (MoE) architectures are designed to handle such diversity by activating only specific sub-modules (experts) for each input token, they face significant challenges in this context.
First, standard MoE models often suffer from expert overlap, where token-level routing leads to experts learning redundant knowledge, thereby diluting specialization. Second, these models typically ignore sequence-level context. By routing solely at the token level, the model may fail to capture the broader intent of a prompt (e.g., identifying a sequence as a coding problem), making it difficult to effectively partition complex data. To address these issues, we introduce Mixture-of-Clustered-Experts (MoCE), a hierarchical routing architecture that respects both sequence-level intent and token-level nuances.
Methods
The core of MoCE is a dual-stage routing mechanism designed to encourage expert specialization through structured input partitioning.
Dual-Stage Routing
- Sequence-Level Group Allocation: The model first generates an embedding of the entire input sequence using a pre-trained encoder (e.g., Instructor or E5). This embedding is then assigned to a specific "Expert Group" using a k-means clustering model. This stage acts as a "prior," pre-sorting the workload based on semantic similarity.
- Token-Level Expert Allocation: Once a group is selected, the model performs standard top-$k$ routing at the token level, but restricted only to the experts within that specific group. This forces experts to specialize in the data types characteristic of their assigned cluster.
To maintain efficiency, MoCE utilizes Parameter-Efficient Sparsity Crafting (PESC), employing lightweight adapters instead of full Feed-Forward Network (FFN) upcycling. Additionally, a variant called MoCE-V incorporates "General Experts"—sub-modules that process every sequence regardless of its cluster—to preserve global knowledge and prevent excessive siloing.
Experimental Setup
The MoCE models were instruction-tuned on a diverse dataset of approximately 1 million samples, combining SlimOrca (518K multi-task reasoning examples), Magicoder (110K coding problems), and MetaMathQA (395K mathematical questions). Performance was evaluated across four major categories: Coding (HumanEval/MBPP), Mathematics (GSM8K/MathQA), General Knowledge (MMLU-Pro/BBH), and Reasoning (ARC/Winogrande).
Key Results and Analysis
MoCE demonstrates significant performance gains over standard adapter-based MoE baselines. For instance, MoCE-E5 achieved an average score of 37.99, compared to 35.62 for the PESC baseline.
Domain-Specific Gains and Generalization
The architecture showed particularly strong improvements in specialized domains. On mathematical tasks (GSM8K), scores rose from 33.21 to 41.93, while coding performance (HumanEval) improved from 16.00 to 19.28. The inclusion of "General Experts" in the MoCE-V variant often yielded the best results, suggesting that a balance between specialized and shared knowledge is optimal for complex instruction tuning.
Furthermore, MoCE proved to be a robust, "plug-and-play" enhancement across different model families. When applied to Mistral-7B and Qwen-7B, the architecture consistently outperformed dense counterparts and other specialized MoE models like DeepSeekMoE, maintaining a better balance between specialized skill and general reasoning.
Conclusion
Mixture-of-Clustered-Experts (MoCE) effectively addresses the limitations of standard MoE in instruction tuning by introducing a hierarchical routing mechanism. By combining sequence-level clustering with token-level routing, MoCE achieves superior expert specialization and generalization. While current implementations focus on adapter-based sparsity due to hardware constraints, future work will explore scaling MoCE to full FFN layers and expanding its application to multilingual and domain-specific contexts.
Summary updated