ZipMoE: A Theoretically-Grounded Mixture of Experts Approach forParameter-Efficient Deep Learning

Lin Chen, Kyriakos Axiotis, Gang Fu, Kaiyuan Wang, Mohammadhossein Bateni, Vahab MirrokniAISTATS 2026 · OpenReview

Abstract

The relentless growth of large language models (LLMs) presents formidable challenges for their training and deployment. To address this critical bottleneck, we introduce ZipMoE, a novel family of parameter-efficient building blocks inspired by the Mixture of Experts (MoE) paradigm. ZipMoE offers a modular and highly efficient substitute for conventional fully connected layers. We provide a rigorous theoretical analysis of ZipMoE's expressiveness, formally demonstrating its superior representational capacity over low-rank factorization. Furthermore, in a least squares regression setting, we prove that ZipMoE achieves a lower test error bound. Our empirical results—featuring comprehensive comparisons against low-rank, Monarch, and Kronecker methods—corroborate these theoretical findings. We demonstrate that ZipMoE consistently attains superior model quality under equivalent parameter or FLOP budgets, establishing it as a potent component for building efficient and powerful deep learning architectures.
BibTeX
@inproceedings{chen2026zipmoe,
  author = {Lin Chen and Kyriakos Axiotis and Gang Fu and Kaiyuan Wang and Mohammadhossein Bateni and Vahab Mirrokni},
  title = {{ZipMoE: A Theoretically-Grounded Mixture of Experts Approach forParameter-Efficient Deep Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=NtcXbO1Qza},
}

Loading related content...

Figure