TL;DR
When compressing Mixture-of-Experts (MoE) models for production, it is significantly better to delete experts (pruning) than to combine them (merging). While academic benchmarks suggest the two methods are neck-and-neck, this study reveals that merging severely damages generative quality and breaks strict output requirements like JSON and HTML formatting far more than simple pruning.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) models, like Mixtral or Qwen, are highly efficient during inference because they only activate a fraction of their parameters for any given token. However, they are "memory hogs"—to run them, you must load the entire model into GPU VRAM. For a 30B-parameter model, this can require over 60GB of memory, driving up production costs.
To solve this, researchers use "one-shot expert compression" to shrink the model by removing or merging experts. Two leading methods have emerged: REAP (which prunes the least important experts) and REAM (which merges them into more important ones). Until now, these have only been compared on academic benchmarks (like multiple-choice questions). This paper asks: how do they perform on the messy, high-stakes tasks of a real-world production environment?
Methods
The authors conducted a head-to-head comparison between:
- REAP (Pruning): This method identifies the "least salient" experts—those that contribute the least to the model's output during a calibration pass—and simply deletes them.
- REAM (Merging): Instead of deleting experts, REAM attempts to preserve their knowledge by aligning and merging them into the most important experts using a technique called Hungarian matching.
The researchers tested these methods on two 30B-parameter models: Qwen3-30B and Nemotron-3.5-Lightning. They evaluated them at 25% and 50% compression levels (removing/merging 32 or 64 out of 128 experts per layer).
Datasets
The study moves beyond standard benchmarks to use five multilingual production tasks from the web-hosting company Hostinger:
- Website Cues: Classification (JSON output).
- Brand Names: Span extraction (JSON output).
- Product Categorization: Open-label categorization (JSON output).
- Domain Descriptions: Short marketing text generation.
- Email Templates: Long-form structured generation (JSON containing complex HTML).
These tasks are summarized in the following table:
The models were calibrated on ~590k tokens of real production conversations, which were predominantly English.
Key Insights and Ideas
- The "Merging" Trap: While merging sounds better in theory (preserving knowledge), it actually introduces "irreducible error." By averaging experts that the model's router originally intended for different inputs, merging creates "muddled" experts that degrade performance, especially in generative tasks.
- Contracts Break Before Content: The most striking finding is that compression doesn't just make the model "dumber"—it makes it "unreliable." Between 90% and 95% of the performance loss in complex tasks (like email generation) came from "hard-gate" failures: invalid JSON, broken HTML layouts, or the model falling into infinite repetition loops.
- Language Confusion: Compressed models often "forget" their language constraints. For example, when asked for a domain description in another language, compressed models frequently started with the English phrase "This domain is..." before switching back to the target language—a failure likely caused by the English-heavy calibration data.
- Benchmarks vs. Reality: On academic benchmarks, REAP and REAM appear to be within one point of each other. In production generation, the gap is massive. REAP at 50% compression actually outperformed REAM at only 25% compression on Qwen3's generative tasks.
Key Results
Pruning (REAP) was the clear winner, proving significantly better than merging (REAM) in 16 out of 20 direct comparisons.
As seen in Figure 1, while both methods hold up well on simple discriminative tasks (left), merging falls off a cliff on generative tasks (right). For Qwen3-30B at 25% compression, REAP retained 93.6% of generative quality, while REAM plummeted to 62.5%.
Figure 2 highlights the magnitude of this lead, particularly in the "Email Templates" task, where REAP outperformed REAM by over 30 rubric points.
Limitations and Future Work
- Private Data: The evaluation uses private production logs, making it impossible for outside researchers to replicate the exact results on the same data.
- Calibration Bias: The calibration data was ~68% English. The authors noted that this likely contributed to the disproportionate performance drop in non-English tasks. Future work should explore more balanced multilingual calibration.
- Model Scope: The study focused on 30B-class models. Whether these findings hold for much smaller (e.g., 8B) or much larger (e.g., 100B+) MoE models remains an open question.
- Complementary Techniques: The authors suggest that expert compression should be studied alongside quantization (e.g., 4-bit weights) to see if the errors compound or cancel out.
Summary updated