MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs

Xiaodong Chen, Mingming Ha, Zhenzhong Lan, Jing Zhang, Jianguo Li

TL;DR

MoBE (Mixture-of-Basis-Experts) is a novel parameter-efficient compression method for Mixture-of-Experts (MoE) large language models. By decomposing expert weights into unique transformation matrices and a shared set of non-linear basis matrices, MoBE reduces total parameter counts by 24% to 30% on trillion-parameter scales with a minimal accuracy drop of just 1% to 2%.

What Problem Does This Paper Address?

The Mixture-of-Experts (MoE) architecture has become the go-to paradigm for scaling large language models (LLMs) like DeepSeek-V3 and Kimi-K2. By activating only a subset of experts per token, MoEs achieve massive capacity with manageable computational costs during inference.

However, MoEs suffer from a massive deployment bottleneck: total parameter size. Even if only a fraction of the model is active at any given moment, the entire model must reside in GPU memory. For trillion-parameter models, this requires prohibitive, high-end hardware clusters.

Prior attempts to compress MoEs generally fall into two camps:

  1. Expert Pruning/Merging: Removing or combining entire experts, which permanently discards specialized knowledge and tanks downstream performance.
  2. Matrix Decomposition (SVD): Compressing expert weights using Singular Value Decomposition. While better than pruning, standard SVD-based approaches still suffer from severe accuracy drops (7%–14% relatively) because they assume the weight matrices are low-rank, which isn't actually true for trained MoE experts.

Methods

MoBE introduces a novel matrix factorization approach tailored specifically to the multi-expert structure of MoE layers. Instead of compressing each expert in isolation, MoBE looks at the commonalities across all experts within a layer.

Specifically, the up and gate projection matrices ($W$) of each expert are factorized as: $$\hat{W}^i = A^i f\left(\sum_{j=1}^m \alpha^{i,j} B^j\right)$$

Here is how it breaks down:

  • Shared Basis Matrices ($B^j$): A small set of $m$ basis matrices are shared across all experts in a given MoE layer to capture common, redundant information.
  • Expert-Specific Coefficients ($\alpha^{i,j}$): Learnable weights that determine how much each expert relies on each basis matrix.
  • Expert-Specific Transformation Matrix ($A^i$): A smaller, unique matrix for each expert that preserves its specialized knowledge.
  • Non-linear Activation ($f$): Crucially, the authors introduce a non-linear activation function (like SiLU or Tanh) after combining the basis matrices. This significantly boosts the expressive power of the compressed representation.

The factorization is optimized layer-by-layer using the Adam optimizer to minimize the Mean Squared Error (MSE) between the original and reconstructed weights. To stabilize this process, the authors apply a Z-score normalization technique across the expert weights before factorization.

Additionally, because the extra matrix multiplications can increase activation memory during inference, the authors propose a variant called $\text{MoBE}^\dagger$, which slightly reduces the number of activated experts (e.g., from $k=8$ to $k'=6$) to maintain computational efficiency.

Datasets

The authors evaluated MoBE across a comprehensive suite of 15 popular downstream benchmarks spanning four primary domains:

  • General Knowledge: MMLU, C-Eval, CMMLU, BBH
  • General Reasoning: ARC-Challenge, IFEval, GPQA
  • Mathematics: Math, GSM8k, AIME24, AIME25
  • Coding: MBPP, HumanEval, LiveCodeBench-v5, MultiPL-E

These benchmarks were tested against several state-of-the-art open-source MoE models, including Ling-Lite-Chat, DeepSeek-V2-Lite-Chat, Qwen3-30B-A3B-2507, Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B), and Kimi-K2-Instruct (1T).

Key Insights and Ideas

  • The SVD Low-Rank Fallacy: The authors analyze the effective rank of expert weight matrices and discover they consistently exceed the compression thresholds required by standard SVD methods. Forcing standard SVD compression eliminates critical high-rank information, explaining why prior methods like MoLAE and $D^2$-MoE suffer from severe performance degradation.
  • The Power of Non-Linearity in Compression: One of the most interesting findings is the impact of the activation function $f$ within the decomposition formula. Using a bipolar activation function like SiLU or Tanh significantly reduces reconstruction error compared to a purely linear combination. Conversely, using ReLU introduces excessive sparsity, which harms performance.
  • Sparsity vs. Total Parameters: As MoE models scale toward the trillion-parameter mark, compressing the total parameter count becomes far more critical for practical deployment than optimizing the activation parameter count.

Key Results

MoBE consistently outperformed existing state-of-the-art compression techniques ($D^2$-MoE and MoLAE) by a wide margin, maintaining much lower reconstruction errors across all layers.

Figure 1: Relative performance comparison of different MoE compression methods. Relative accuracy is the ratio of the compressed model’s performance to that of the original model. The accuracy are averaged over 15 benchmarks as shown in Table 3. Applying D2-MoE to large models like Qwen3-235B-A22B-2507, DeepSeek- V3-0324 and Kimi-K2-Instruct is computationally prohibitive on an 8x H100 GPU machine; therefore, it is excluded from these comparisons. MoBE is evaluated at compression rates similar to or higher than the baseline methods (MoLAE, D2-MoE). Absolute performance is detailed in Appendix A (Figure 8).
  • Massive Parameter Savings: MoBE successfully reduced the total parameter counts of Qwen3-235B, DeepSeek-V3 (671B), and Kimi-K2-Instruct (1T) by 24% to 30%.
  • Minimal Accuracy Drops: While baseline compression methods caused performance to plummet by 7% to 15%, MoBE retained up to 98% of the original model performance, incurring only a minor 1% to 2% absolute accuracy drop across the 15 benchmarks.
  • Reconstruction Accuracy: Layer-by-layer analysis showed that MoBE achieved over a 50% reduction in Mean Squared Error (MSE) compared to MoLAE and $D^2$-MoE.
Figure 2: Comparison of pre-layer MSE for compressing the gate (a) and up (b) matrices of Ling-Lite-Chat using MoBE, D2-MoE and MoLAE.

Limitations and Future Work

Despite its strong performance, the authors note two primary limitations:

  1. Minor Performance Gap: There is still a small accuracy drop compared to the uncompressed base models. The authors suggest that incorporating full-network knowledge distillation (KD) during or after the factorization process could bridge this remaining gap.
  2. Inference Efficiency: Currently, implementing MoBE requires making multiple sequential calls to standard fused-MoE kernels to mimic the factorization math. This introduces overhead. To fully unlock the speed and efficiency benefits of MoBE, future work will require engineering a custom "mega-kernel" that natively handles the entire MoBE factorization in a single pass.

Summary updated

Abstract

The Mixture-of-Experts (MoE) architecture has become a predominant paradigm for scaling large language models (LLMs). Despite offering strong performance and computational efficiency, large MoE-based LLMs like DeepSeek-V3-0324 and Kimi-K2-Instruct present serious challenges due to substantial memory requirements in deployment. While recent works have explored MoE compression to address this issue, existing methods often suffer from considerable accuracy drops (e.g., 7-14% relatively) even at modest compression rates. This paper introduces a novel Mixture-of-Basis-Experts (MoBE) method that achieves model compression while incurring minimal accuracy drops. Specifically, each up/gate matrix in an expert is decomposed via a rank decomposition as W = AB, where matrix A is unique to each expert. The relatively larger matrix B is further re-parameterized as a linear combination of basis matrices {Bi} shared across all experts within a given MoE layer. The factorization is learned by minimizing the reconstruction error relative to the original weight matrices. Experiments demonstrate that MoBE achieves notably lower accuracy drops compared to prior works. For instance, MoBE can reduce the parameter counts of Qwen3-235B-A22B-2507, DeepSeek-V3-0324 (671B) and Kimi-K2-Instruct (1T) by 24%-30% with only 1%-2% accuracy drop (about 2% drops when measured relatively).
BibTeX
@inproceedings{chen2026mobe,
  author = {Xiaodong Chen and Mingming Ha and Zhenzhong Lan and Jing Zhang and Jianguo Li},
  title = {{MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=8RV6H50OSf},
}
Figure