ZipMoE: A Theoretically-Grounded Mixture of Experts Approach forParameter-Efficient Deep Learning
Lists
Only you see this
Abstract
The relentless growth of large language models (LLMs) presents formidable challenges for their training and deployment. To address this critical bottleneck, we introduce ZipMoE, a novel family of parameter-efficient building blocks inspired by the Mixture of Experts (MoE) paradigm. ZipMoE offers a modular and highly efficient substitute for conventional fully connected layers. We provide a rigorous theoretical analysis of ZipMoE's expressiveness, formally demonstrating its superior representational capacity over low-rank factorization. Furthermore, in a least squares regression setting, we prove that ZipMoE achieves a lower test error bound. Our empirical results—featuring comprehensive comparisons against low-rank, Monarch, and Kronecker methods—corroborate these theoretical findings. We demonstrate that ZipMoE consistently attains superior model quality under equivalent parameter or FLOP budgets, establishing it as a potent component for building efficient and powerful deep learning architectures.
BibTeX
@inproceedings{chen2026zipmoe,
author = {Lin Chen and Kyriakos Axiotis and Gang Fu and Kaiyuan Wang and Mohammadhossein Bateni and Vahab Mirrokni},
title = {{ZipMoE: A Theoretically-Grounded Mixture of Experts Approach forParameter-Efficient Deep Learning}},
booktitle = {International Conference on Artificial Intelligence and Statistics},
year = {2026},
url = {https://openreview.net/forum?id=NtcXbO1Qza},
}Loading related content...