Abstract
As revealed by the scaling law of fine-grained Mixture-of-Experts (MoE), model performance ceases to improve once the granularity of the intermediate dimension exceeds an optimal threshold. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions. By enhancing expert specialization beyond the single-dimension limit, FineRMoE achieves state-of-the-art performance. We further introduce a bi-level sparse forward computation paradigm, a specialized routing mechanism, and a generalized upcycling method to build FineRMoE cost-effectively from pre-trained dense models. Extensive experiments demonstrate that FineRMoE achieves 6× higher parameter efficiency and massive inference speedups—up to 281× lower prefill latency and 136× higher decoding throughput—compared to strong baselines.
Introduction
Mixture-of-Experts (MoE) models have become a cornerstone for scaling Large Language Models (LLMs) efficiently. Recent "fine-grained MoE" approaches split the intermediate dimension of Feed-Forward Networks (FFNs) into numerous small experts. However, research indicates a performance plateau when granularity is increased in only one dimension. FineRMoE breaks this ceiling by introducing "multi-dimensional" fine-grained experts, constraining both intermediate and output dimensions to encourage deeper specialization and reduce redundancy, similar to the multi-head design in attention mechanisms.
Architecture and Methodology
The core of FineRMoE is its Bi-Level Sparse Architecture, which moves beyond the traditional weighted sum of experts.
As illustrated in Figure 1, the architecture consists of:
- Sparse Sum Layer: Fine-grained experts are grouped and summed to produce dimension-reduced intermediate vectors.
- Sparse Concatenation Layer: These vectors are then concatenated to reconstruct the full hidden dimension of the model.
- Specialized Routing: A single router network manages the activation of experts in the sum layer and the selection of groups in the concatenation layer, ensuring efficiency and avoiding the conflicts of multiple independent routers.
To make this architecture practical, the authors developed a Generalized Upcycling method. This allows for the transformation of a pre-trained dense model into a FineRMoE structure by partitioning weights, followed by continued training on a relatively small corpus (e.g., 50 billion tokens), significantly reducing the computational cost compared to training from scratch.
Experimental Results
The researchers evaluated FineRMoE using the Qwen2.5 model family (0.5B, 1.5B, and 7B) across ten standard benchmarks, including MMLU, GSM8K, and MBPP.
As shown in Figure 2, FineRMoE consistently outperforms both the original pre-trained (PT) dense models and various MoE upcycling baselines (like C32A2 and NVShard). Notably:
- Superior Accuracy: On the 7B scale, FineRMoE achieved an average benchmark score of 70.04, the highest among all tested variants.
- Efficiency and Speed: FineRMoE demonstrates a 6× improvement in parameter efficiency. In inference tests on Qwen2.5-7B, it achieved a 281× reduction in Time to First Token (TTFT) and a 136× increase in decoding throughput compared to the C32A2 baseline.
Conclusion
FineRMoE represents a significant step forward in MoE design by extending fine-grained specialization to the output dimension. By combining this bi-level architecture with an efficient upcycling strategy, it provides a path to scaling LLMs that is both high-performing and computationally efficient. Future work will explore the impact of training FineRMoE from scratch on larger datasets and the development of specialized hardware kernels to further optimize its massive inference advantages.
Summary updated