Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks

Tomas Rasymas, Povilas KvedarasarXiv 2026 · arXiv

TL;DR

When compressing Mixture-of-Experts (MoE) models for production, it is significantly better to delete experts (pruning) than to combine them (merging). While academic benchmarks suggest the two methods are neck-and-neck, this study reveals that merging severely damages generative quality and breaks strict output requirements like JSON and HTML formatting far more than simple pruning.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models, like Mixtral or Qwen, are highly efficient during inference because they only activate a fraction of their parameters for any given token. However, they are "memory hogs"—to run them, you must load the entire model into GPU VRAM. For a 30B-parameter model, this can require over 60GB of memory, driving up production costs.

To solve this, researchers use "one-shot expert compression" to shrink the model by removing or merging experts. Two leading methods have emerged: REAP (which prunes the least important experts) and REAM (which merges them into more important ones). Until now, these have only been compared on academic benchmarks (like multiple-choice questions). This paper asks: how do they perform on the messy, high-stakes tasks of a real-world production environment?

Methods

The authors conducted a head-to-head comparison between:

  • REAP (Pruning): This method identifies the "least salient" experts—those that contribute the least to the model's output during a calibration pass—and simply deletes them.
  • REAM (Merging): Instead of deleting experts, REAM attempts to preserve their knowledge by aligning and merging them into the most important experts using a technique called Hungarian matching.

The researchers tested these methods on two 30B-parameter models: Qwen3-30B and Nemotron-3.5-Lightning. They evaluated them at 25% and 50% compression levels (removing/merging 32 or 64 out of 128 experts per layer).

Datasets

The study moves beyond standard benchmarks to use five multilingual production tasks from the web-hosting company Hostinger:

  1. Website Cues: Classification (JSON output).
  2. Brand Names: Span extraction (JSON output).
  3. Product Categorization: Open-label categorization (JSON output).
  4. Domain Descriptions: Short marketing text generation.
  5. Email Templates: Long-form structured generation (JSON containing complex HTML).

These tasks are summarized in the following table:

Table 1: The five production tasks. Lang. = number of languages in the 2,000 validation rows. Tokens = median answer length of the base Qwen3-30B.

The models were calibrated on ~590k tokens of real production conversations, which were predominantly English.

Key Insights and Ideas

  • The "Merging" Trap: While merging sounds better in theory (preserving knowledge), it actually introduces "irreducible error." By averaging experts that the model's router originally intended for different inputs, merging creates "muddled" experts that degrade performance, especially in generative tasks.
  • Contracts Break Before Content: The most striking finding is that compression doesn't just make the model "dumber"—it makes it "unreliable." Between 90% and 95% of the performance loss in complex tasks (like email generation) came from "hard-gate" failures: invalid JSON, broken HTML layouts, or the model falling into infinite repetition loops.
  • Language Confusion: Compressed models often "forget" their language constraints. For example, when asked for a domain description in another language, compressed models frequently started with the English phrase "This domain is..." before switching back to the target language—a failure likely caused by the English-heavy calibration data.
  • Benchmarks vs. Reality: On academic benchmarks, REAP and REAM appear to be within one point of each other. In production generation, the gap is massive. REAP at 50% compression actually outperformed REAM at only 25% compression on Qwen3's generative tasks.

Key Results

Pruning (REAP) was the clear winner, proving significantly better than merging (REAM) in 16 out of 20 direct comparisons.

Figure 1: Comparison of score retention between pruning (REAP) and merging (REAM).

As seen in Figure 1, while both methods hold up well on simple discriminative tasks (left), merging falls off a cliff on generative tasks (right). For Qwen3-30B at 25% compression, REAP retained 93.6% of generative quality, while REAM plummeted to 62.5%.

Figure 2: Head-to-head metric difference (REAP - REAM). Positive values favor pruning.

Figure 2 highlights the magnitude of this lead, particularly in the "Email Templates" task, where REAP outperformed REAM by over 30 rubric points.

Limitations and Future Work

  • Private Data: The evaluation uses private production logs, making it impossible for outside researchers to replicate the exact results on the same data.
  • Calibration Bias: The calibration data was ~68% English. The authors noted that this likely contributed to the disproportionate performance drop in non-English tasks. Future work should explore more balanced multilingual calibration.
  • Model Scope: The study focused on 30B-class models. Whether these findings hold for much smaller (e.g., 8B) or much larger (e.g., 100B+) MoE models remains an open question.
  • Complementary Techniques: The authors suggest that expert compression should be studied alongside quantization (e.g., 4-bit weights) to see if the errors compound or cancel out.

Summary updated

Abstract

Mixture-of-Experts language models are cheap to run per token but expensive to hold in GPU memory, which drives the cost of serving them in production. One-shot expert compression shrinks them after a single calibration pass. Its two current state-of-the-art methods are REAP for expert pruning, which deletes the least salient experts, and REAM for expert merging, which merges them into the most salient ones. Both have been evaluated only on academic benchmarks. We compare the two on five multilingual production tasks of a web-hosting company, from classification to the generation of marketing HTML email templates, with two 30B-parameter models (Qwen3-30B-A3B-Instruct-2507 and the hybrid Mamba-attention Nemotron-3.5-Lightning) at 25% and 50% of experts removed and the same calibration data. Pruning wins: on the same rows, REAP is significantly better in 16 of 20 comparisons and REAM in 2. At 25%, REAP keeps 93.6% of Qwen3-30B's generative quality and REAM 62.5%, although on benchmarks the two are within a point for the same model. The damage concentrates in the output contract rather than the content: 89.8-95.2% of each significant loss on email templates comes from hard-gate failures such as invalid JSON, broken mobile layouts and runaway generation, and most failed non-English domain descriptions copy the prompt's English opening phrase. Cross-perplexity and logit divergence, two diagnostics from the REAP paper, also track production quality, as does the rate of repetition loops; n-gram diversity does not. We close with practical guidance for compressing MoE models for production.
BibTeX
@misc{rasymas2026prune,
  author = {Tomas Rasymas and Povilas Kvedaras},
  title = {{Prune, Don't Merge: Expert Compression of Mixture-of-Experts LLMs on Multilingual Production Tasks}},
  year = {2026},
  eprint = {2611.12240},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12240},
}
Figure