FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

Ning Liao, Xiaoxing Wang, Xiaohan Qin, Junchi Yan

Abstract

As revealed by the scaling law of fine-grained Mixture-of-Experts (MoE), model performance ceases to improve once the granularity of the intermediate dimension exceeds an optimal threshold. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions. By enhancing expert specialization beyond the single-dimension limit, FineRMoE achieves state-of-the-art performance. We further introduce a bi-level sparse forward computation paradigm, a specialized routing mechanism, and a generalized upcycling method to build FineRMoE cost-effectively from pre-trained dense models. Extensive experiments demonstrate that FineRMoE achieves 6× higher parameter efficiency and massive inference speedups—up to 281× lower prefill latency and 136× higher decoding throughput—compared to strong baselines.

Introduction

Mixture-of-Experts (MoE) models have become a cornerstone for scaling Large Language Models (LLMs) efficiently. Recent "fine-grained MoE" approaches split the intermediate dimension of Feed-Forward Networks (FFNs) into numerous small experts. However, research indicates a performance plateau when granularity is increased in only one dimension. FineRMoE breaks this ceiling by introducing "multi-dimensional" fine-grained experts, constraining both intermediate and output dimensions to encourage deeper specialization and reduce redundancy, similar to the multi-head design in attention mechanisms.

Architecture and Methodology

The core of FineRMoE is its Bi-Level Sparse Architecture, which moves beyond the traditional weighted sum of experts.

The FineRMoE architecture featuring bi-level sparse computation layers and a specialized routing mechanism.

As illustrated in Figure 1, the architecture consists of:

  1. Sparse Sum Layer: Fine-grained experts are grouped and summed to produce dimension-reduced intermediate vectors.
  2. Sparse Concatenation Layer: These vectors are then concatenated to reconstruct the full hidden dimension of the model.
  3. Specialized Routing: A single router network manages the activation of experts in the sum layer and the selection of groups in the concatenation layer, ensuring efficiency and avoiding the conflicts of multiple independent routers.

To make this architecture practical, the authors developed a Generalized Upcycling method. This allows for the transformation of a pre-trained dense model into a FineRMoE structure by partitioning weights, followed by continued training on a relatively small corpus (e.g., 50 billion tokens), significantly reducing the computational cost compared to training from scratch.

Experimental Results

The researchers evaluated FineRMoE using the Qwen2.5 model family (0.5B, 1.5B, and 7B) across ten standard benchmarks, including MMLU, GSM8K, and MBPP.

Performance comparison of FineRMoE against various baselines across different model scales and benchmarks.

As shown in Figure 2, FineRMoE consistently outperforms both the original pre-trained (PT) dense models and various MoE upcycling baselines (like C32A2 and NVShard). Notably:

  • Superior Accuracy: On the 7B scale, FineRMoE achieved an average benchmark score of 70.04, the highest among all tested variants.
  • Efficiency and Speed: FineRMoE demonstrates a 6× improvement in parameter efficiency. In inference tests on Qwen2.5-7B, it achieved a 281× reduction in Time to First Token (TTFT) and a 136× increase in decoding throughput compared to the C32A2 baseline.

Conclusion

FineRMoE represents a significant step forward in MoE design by extending fine-grained specialization to the output dimension. By combining this bi-level architecture with an efficient upcycling strategy, it provides a path to scaling LLMs that is both high-performing and computationally efficient. Future work will explore the impact of training FineRMoE from scratch on larger datasets and the development of specialized hardware kernels to further optimize its massive inference advantages.

Summary updated

Abstract

As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
BibTeX
@misc{liao2026finermoe,
  author = {Ning Liao and Xiaoxing Wang and Xiaohan Qin and Junchi Yan},
  title = {{FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach}},
  year = {2026},
  eprint = {2603.13364},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.13364},
}
Figure