TL;DR
MoBE (Mixture-of-Basis-Experts) is a novel parameter-efficient compression method for Mixture-of-Experts (MoE) large language models. By decomposing expert weights into unique transformation matrices and a shared set of non-linear basis matrices, MoBE reduces total parameter counts by 24% to 30% on trillion-parameter scales with a minimal accuracy drop of just 1% to 2%.
What Problem Does This Paper Address?
The Mixture-of-Experts (MoE) architecture has become the go-to paradigm for scaling large language models (LLMs) like DeepSeek-V3 and Kimi-K2. By activating only a subset of experts per token, MoEs achieve massive capacity with manageable computational costs during inference.
However, MoEs suffer from a massive deployment bottleneck: total parameter size. Even if only a fraction of the model is active at any given moment, the entire model must reside in GPU memory. For trillion-parameter models, this requires prohibitive, high-end hardware clusters.
Prior attempts to compress MoEs generally fall into two camps:
- Expert Pruning/Merging: Removing or combining entire experts, which permanently discards specialized knowledge and tanks downstream performance.
- Matrix Decomposition (SVD): Compressing expert weights using Singular Value Decomposition. While better than pruning, standard SVD-based approaches still suffer from severe accuracy drops (7%–14% relatively) because they assume the weight matrices are low-rank, which isn't actually true for trained MoE experts.
Methods
MoBE introduces a novel matrix factorization approach tailored specifically to the multi-expert structure of MoE layers. Instead of compressing each expert in isolation, MoBE looks at the commonalities across all experts within a layer.
Specifically, the up and gate projection matrices ($W$) of each expert are factorized as: $$\hat{W}^i = A^i f\left(\sum_{j=1}^m \alpha^{i,j} B^j\right)$$
Here is how it breaks down:
- Shared Basis Matrices ($B^j$): A small set of $m$ basis matrices are shared across all experts in a given MoE layer to capture common, redundant information.
- Expert-Specific Coefficients ($\alpha^{i,j}$): Learnable weights that determine how much each expert relies on each basis matrix.
- Expert-Specific Transformation Matrix ($A^i$): A smaller, unique matrix for each expert that preserves its specialized knowledge.
- Non-linear Activation ($f$): Crucially, the authors introduce a non-linear activation function (like SiLU or Tanh) after combining the basis matrices. This significantly boosts the expressive power of the compressed representation.
The factorization is optimized layer-by-layer using the Adam optimizer to minimize the Mean Squared Error (MSE) between the original and reconstructed weights. To stabilize this process, the authors apply a Z-score normalization technique across the expert weights before factorization.
Additionally, because the extra matrix multiplications can increase activation memory during inference, the authors propose a variant called $\text{MoBE}^\dagger$, which slightly reduces the number of activated experts (e.g., from $k=8$ to $k'=6$) to maintain computational efficiency.
Datasets
The authors evaluated MoBE across a comprehensive suite of 15 popular downstream benchmarks spanning four primary domains:
- General Knowledge: MMLU, C-Eval, CMMLU, BBH
- General Reasoning: ARC-Challenge, IFEval, GPQA
- Mathematics: Math, GSM8k, AIME24, AIME25
- Coding: MBPP, HumanEval, LiveCodeBench-v5, MultiPL-E
These benchmarks were tested against several state-of-the-art open-source MoE models, including Ling-Lite-Chat, DeepSeek-V2-Lite-Chat, Qwen3-30B-A3B-2507, Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B), and Kimi-K2-Instruct (1T).
Key Insights and Ideas
- The SVD Low-Rank Fallacy: The authors analyze the effective rank of expert weight matrices and discover they consistently exceed the compression thresholds required by standard SVD methods. Forcing standard SVD compression eliminates critical high-rank information, explaining why prior methods like MoLAE and $D^2$-MoE suffer from severe performance degradation.
- The Power of Non-Linearity in Compression: One of the most interesting findings is the impact of the activation function $f$ within the decomposition formula. Using a bipolar activation function like SiLU or Tanh significantly reduces reconstruction error compared to a purely linear combination. Conversely, using ReLU introduces excessive sparsity, which harms performance.
- Sparsity vs. Total Parameters: As MoE models scale toward the trillion-parameter mark, compressing the total parameter count becomes far more critical for practical deployment than optimizing the activation parameter count.
Key Results
MoBE consistently outperformed existing state-of-the-art compression techniques ($D^2$-MoE and MoLAE) by a wide margin, maintaining much lower reconstruction errors across all layers.
- Massive Parameter Savings: MoBE successfully reduced the total parameter counts of Qwen3-235B, DeepSeek-V3 (671B), and Kimi-K2-Instruct (1T) by 24% to 30%.
- Minimal Accuracy Drops: While baseline compression methods caused performance to plummet by 7% to 15%, MoBE retained up to 98% of the original model performance, incurring only a minor 1% to 2% absolute accuracy drop across the 15 benchmarks.
- Reconstruction Accuracy: Layer-by-layer analysis showed that MoBE achieved over a 50% reduction in Mean Squared Error (MSE) compared to MoLAE and $D^2$-MoE.
Limitations and Future Work
Despite its strong performance, the authors note two primary limitations:
- Minor Performance Gap: There is still a small accuracy drop compared to the uncompressed base models. The authors suggest that incorporating full-network knowledge distillation (KD) during or after the factorization process could bridge this remaining gap.
- Inference Efficiency: Currently, implementing MoBE requires making multiple sequential calls to standard fused-MoE kernels to mimic the factorization math. This introduces overhead. To fully unlock the speed and efficiency benefits of MoBE, future work will require engineering a custom "mega-kernel" that natively handles the entire MoBE factorization in a single pass.
Summary updated