TL;DR
This paper introduces Mixture-of-Clustered-Experts (MoCE), a novel architecture that enhances Mixture-of-Experts (MoE) models for instruction tuning by employing a two-stage routing mechanism. MoCE first groups experts based on sequence-level input features and then routes tokens within those groups, leading to improved expert specialization and better generalization across diverse tasks.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) models have shown great promise in scaling up large language models (LLMs) by allowing conditional computation, meaning only a subset of parameters are activated for any given input. This is particularly beneficial for instruction tuning, where models are trained on a vast array of tasks and domains. However, standard MoE architectures struggle with the inherent heterogeneity of instruction tuning data. This leads to a lack of expert specialization, where individual experts don't develop distinct knowledge, resulting in redundant computations and suboptimal performance. Furthermore, traditional MoE routing operates solely at the token level, indirectly capturing sequence-level information like domain or task, which limits its ability to effectively partition inputs based on their knowledge requirements. This paper aims to address these limitations by proposing a hierarchical routing mechanism that explicitly leverages both sequence-level and token-level information to foster better expert specialization and generalization.
Methods
The core innovation of this paper is the Mixture-of-Clustered-Experts (MoCE) architecture, which introduces a dual-stage routing mechanism.
- Sequence-Level Expert Group Allocation:
- Instead of directly routing tokens to experts, MoCE first groups experts together.
- It then uses sequence embeddings (derived from models like Instructor or E5) to predict which expert group is most suitable for a given input sequence.
- This is achieved using a k-means clustering algorithm applied to the sequence embeddings. The algorithm partitions inputs into clusters, and each cluster is mapped to a specific expert group. This allows for a coarse-grained segmentation of inputs based on their overall characteristics.
- Token-Level Expert Allocation:
- Once an expert group is activated for a sequence, the standard token-level routing mechanism is applied, but only within that activated group.
- A group-specific gating function then selects the top-k experts from the group to process each token.
- This hierarchical approach ensures that inputs are first directed to a relevant set of experts (grouping) and then efficiently processed by the most appropriate experts within that group (token-level routing).
To address the high memory requirements of MoE models, MoCE adopts a parameter-efficient adapter-based approach, integrating adapters into each expert's feed-forward network (FFN). This allows for efficient fine-tuning without loading all experts into memory simultaneously.
Datasets
The paper uses three primary datasets for instruction tuning:
- SlimOrca: A curated dataset of 518K multi-task examples.
- Magicoder: A dataset of 110K coding problems.
- MetaMathQA: A dataset of 395K mathematical questions.
In total, approximately 1 million data points are used for training. For evaluation, they employ a comprehensive set of benchmarks across four categories: Mathematics (MathQA, GSM8K), Code (HumanEval, MBPP), Knowledge & Reasoning (BBH, MMLU-Pro), and Reasoning (Winogrande, ARC).
Key Insights and Ideas
- Hierarchical Routing is Key: The paper strongly suggests that a simple token-level router is insufficient for heterogeneous instruction tuning data. Combining sequence-level grouping with token-level routing is crucial for effective specialization.
- Clustering for Specialization: Using k-means clustering on sequence embeddings effectively partitions inputs based on their underlying characteristics, enabling expert groups to develop more specialized knowledge.
- Domain-Specific Preferences: The cluster predictions show distinct preferences for different domains (e.g., MBPP inputs clustering differently than ARC inputs), highlighting how MoCE can learn to assign tasks to appropriate expert groups.
- General Experts Enhance Collaboration: A variant of MoCE that includes general experts alongside specialized ones shows improved performance, suggesting that a balance between broad knowledge and specialized expertise is beneficial.
- Adapter-Based MoE is Practical: The use of adapters makes MoCE more memory-efficient and practical for real-world applications, demonstrating that advanced MoE architectures can be deployed without prohibitive hardware costs.
Key Results
- MoCE consistently outperforms baseline models (including adapter-based LLaMA, BTX, and domain-specific PESC models) across various benchmarks, achieving an average score of 37.99 with MoCE-E5 and 37.61 with MoCE-Instructor.
- MoCE shows significant improvements in mathematics and code domains, with MoCE-E5 outperforming PESC by +8.72 on GSM8K and +3.28 on HumanEval.
- The model demonstrates strong generalization capabilities, matching or surpassing in-domain performance while also performing well on out-of-domain tasks, unlike domain-specialized models.
- MoCE achieves substantial performance improvements over dense models of similar size, for instance, increasing GSM8K accuracy from 42.76 to 54.97 for Mistral 7B.
- Ablation studies confirm the importance of both the sequence-level clustering and token-level routing stages, with removing either leading to performance degradation.
Limitations and Future Work
- Computational Constraints: The current implementation of MoCE uses adapter-based models for efficiency. Applying the dual-stage routing directly to full FFN layers in instruction tuning is left as future work.
- Interpretability: While k-means clustering is used, the inherent explainability challenges of MoE architectures remain. Further work is needed to address these issues.
- Multilingual and Domain Expansion: The current evaluation focuses on English and specific domains. Extending MoCE to more languages and specialized domains is a promising direction for future research.
Summary updated