MergeMoE: Efficient Compression of MoE Models via Expert Output Merging

Ruijie Miao, Yilun Yao, Zihan Wang, Zhiming Wang, Bairen Yi, LingJun Liu, Yikai Zhao, Tong YangSubmission to ICLR 2026 · Withdrawn

TL;DR

MergeMoE rethinks how to compress Mixture-of-Experts (MoE) models by framing expert merging as an output-approximation problem rather than simple parameter averaging. By decoupling the SwiGLU expert computation and solving for compression matrices using closed-form linear least squares, MergeMoE consistently outperforms prior expert-merging baselines across multiple open-source MoE models without retraining.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures achieve high parameter efficiency by routing each input token to a sparse subset of feed-forward "experts." While this drastically reduces computational FLOPs during inference, it requires storing billions of parameters in VRAM, making deployment difficult on memory-constrained hardware.

Prior MoE compression methods like M-SMoE group experts and merge them by heuristically computing weighted averages of their weight matrices. However, parameter-space averaging lacks a firm theoretical foundation and fails to account for the non-linear transformations inside expert layers, leading to significant performance degradation.

Methods

MergeMoE shifts the perspective from merging expert parameters to merging expert outputs:

  1. Theoretical Formulation: The forward pass of an MoE layer is rewritten using routing and clustering assignment matrices ($A$ and $B$). A merged expert $E'_i$ is designed to approximate a linear combination of original expert outputs $\sum_j B_{ji} E_j(X)$.
  2. Dimension-Reduction Decomposition: For SwiGLU experts ($E(X) = W_D(\sigma(W_G X) \odot W_U X)$), preserving output equality without reducing dimensions would blow up the parameter size. MergeMoE introduces three dimension-reduction matrices ($T_1, T_2, T_3$) to map concatenated expert matrices back to standard expert dimensions.
  3. Decoupled Optimization:
  4. Gating and Up-projections ($T_2, T_3$): Because non-linear activations and Hadamard products make joint optimization intractable, $T_2$ and $T_3$ perform weighted averaging based on expert usage frequency. The authors mathematically prove (Theorem 1) that weighting by relative frequency is optimal for minimizing output error.
  5. Down-projection ($T_1$): Given a small set of calibration inputs $\hat{X}$, the authors formulate the recovery of $T_1$ as a linear least-squares problem, deriving a closed-form solution via the Moore-Penrose pseudoinverse ($T_1 = QP^\dagger$).
  6. Execution Pipeline: Experts are clustered by the cosine similarity of their concatenated $[W_U, W_G]$ matrices around the most frequently activated experts. Merging is performed layer-by-layer backwards (from output to input) on a single GPU using intermediate activation hooks.

Datasets

The authors evaluate MergeMoE on seven standard NLP benchmarks:

  • WinoGrande (coreference resolution)
  • ARC Easy & ARC Challenge (scientific reasoning)
  • HellaSwag (commonsense reasoning)
  • PIQA (physical interaction QA)
  • SQuAD (extractive question answering)
  • MRPC (paraphrase identification)
  • IFEval (instruction following, paired with ShareGPT for optional distillation)

Note: No new datasets were introduced.

Key Insights and Ideas

  • Outputs matter more than weights: Directly averaging weights ignores the non-linear interaction between SwiGLU gating and up-projection. Viewing merging as function approximation in the output space enables rigorous mathematical formulation.
  • Frequency weighting is provably optimal: The paper provides formal theoretical backing showing that under output-error bounds, weighting merged experts by activation frequency minimizes the Quasi-Frobenius error bound.
  • Closed-form calibration rescues performance: Optimizing the down-projection matrix $W_D T_1$ with a handful of forward-pass activations acts as an error corrector for the non-linear activations in $W_G$ and $W_U$.
  • Layer distribution vs. depth: Compressing fewer experts across more layers preserves downstream reasoning capability significantly better than heavily collapsing experts within a few layers.

Key Results

Experiments evaluated DeepSeekMoE (16B $\rightarrow$ 12B), Qwen1.5-MoE-A2.7B (14B $\rightarrow$ 10B), and Qwen3-30B-A3B (30B $\rightarrow$ 25B) against baselines like Average, ZipIt, and M-SMoE:

  • Qwen1.5-MoE: MergeMoE outperforms the previous state-of-the-art (M-SMoE) by +1.50 on WinoGrande (70.48 vs. 68.98), +2.71 on PIQA (79.27 vs. 76.56 / ZipIt 77.80), and +2.45 on MRPC (74.75 vs. 72.30).
  • Qwen3-30B: MergeMoE achieves 83.04 on ARC-e, 63.48 on ARC-c, and 74.93 on HellaSwag—narrowing the gap to the uncompressed model (84.89, 67.49, 76.38) and comfortably beating the Qwen3-4B dense baseline.
  • Low Overhead: Merging layers on Qwen1.5 takes less than 60 seconds on a single NVIDIA H20 GPU.

Limitations and Future Work

  • Sample Size Sensitivity: MergeMoE requires calibration data. Performance collapses to random guessing if fewer than ~32–36 samples are used, though accuracy stabilizes and scales smoothly beyond 40–64 samples.
  • Cross-Task Transfer Gap: Using calibration samples from a single dataset yields minor performance degradation when evaluating on out-of-domain tasks compared to task-specific calibration.
  • Rare Expert Compression: Because the weighting scheme favors frequently activated experts, rarely used experts in the long-tail distribution risk being disproportionately eroded.

Summary updated

Abstract

The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.
BibTeX
@misc{miao2025mergemoe,
  author = {Ruijie Miao and Yilun Yao and Zihan Wang and Zhiming Wang and Bairen Yi and LingJun Liu and Yikai Zhao and Tong Yang},
  title = {{MergeMoE: Efficient Compression of MoE Models via Expert Output Merging}},
  year = {2025},
  eprint = {2510.14436},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14436},
}
Figure