The promise of Mixture-of-Experts (MoE) models lies in their efficiency: they have massive parameter counts but only activate a small fraction of "experts" for any given token. However, storing these models is still expensive. To solve this, researchers use expert pruning—deleting the experts that seem "unimportant" based on a small calibration dataset.
A new paper, "English Calibration Erodes Lithuanian in Expert-Pruned Mixture-of-Experts Language Models," reveals a major pitfall in this process. While pruning based on English data works fine for English, it can catastrophically "break" the model for lower-resource languages like Lithuanian.
TL;DR
Pruning MoE models using only English calibration data severely degrades performance in lower-resource languages, even when English performance remains stable. By simply including target-language data during the pruning process, researchers can preserve multilingual capabilities with almost no cost to English performance.
What Problem Does This Paper Address?
When we prune an MoE model, we use a "calibration set" to see which experts the model actually uses. Standard practice is to use English data, under the assumption that what’s important for English is important for everything else. Previous studies on high-resource languages (like Chinese) suggested this transfer works well.
The authors test this assumption on Lithuanian, a language with complex morphology and much less web presence than English. They wanted to know if the "English-only" pruning strategy creates a "tax" on lower-resource languages that goes unnoticed if you only evaluate in English.
Methods
The researchers pruned four state-of-the-art MoE models (Qwen3-30B, Qwen3.6-35B, Gemma 4, and GLM-4.7-Flash) using REAP (Router-weighted Expert Activation Pruning). REAP scores experts based on how much the model's "router" trusts them and the magnitude of their output.
They tested three calibration strategies:
- English only (EN)
- Lithuanian only (LT)
- A 50/50 mix (MIX)
They removed either 25% or 50% of the experts in every layer and evaluated the resulting compressed models on reading comprehension, translation, and language modeling.
Datasets
- Calibration: Wikipedia (encyclopedic text) and Europarl (parallel parliamentary proceedings).
- Evaluation:
- Belebele: A multilingual reading comprehension benchmark.
- FLORES-200: Used for machine translation (English ↔ Lithuanian).
- Wikipedia: Used to measure "bits per byte" (language modeling accuracy) on held-out articles.
Key Insights and Ideas
- Asymmetric Transfer: This is the "aha moment." Lithuanian calibration preserves English performance remarkably well, but English calibration destroys Lithuanian performance.
- Routing Concentration: Why does this happen? The authors found that Lithuanian is "pickier" than English. In the final layers of the network, Lithuanian routing is highly concentrated: 55–72% of its selections go to just eight experts. English routing is much more distributed. English calibration often flags these "Lithuanian-critical" experts as unimportant and deletes them.
- The "Invisible" Damage: At 25% pruning, a model calibrated on English might lose only 1.7 points in English comprehension but a staggering 21.4 points in Lithuanian. If you only test in English, you’d think the compression was a success, while the model has actually become useless for millions of speakers.
Key Results
- Lithuanian Recovery: Switching to Lithuanian or Mixed calibration removed most of the performance gap. For example, the Lithuanian Belebele loss dropped from ~21 points to just ~3 points.
- Failure Modes: Models pruned with English-only data didn't just get "worse"—they failed in weird ways. They often failed to stop generating (hitting token limits), switched to English mid-sentence, or hallucinated non-existent Lithuanian word forms.
- Domain vs. Language: Interestingly, the language of the calibration data mattered far more than the domain (Wikipedia vs. Europarl).
Limitations and Future Work
The study focused specifically on Lithuanian. While the authors suspect these findings apply to other lower-resource languages, they haven't tested them yet. Additionally, the study used a fixed token budget for calibration; because Lithuanian is less token-efficient than English, the Lithuanian calibration sets actually contained less "text" than the English ones, making the recovery results even more impressive.
Future work could explore non-uniform pruning (removing more experts from middle layers where languages overlap more) or expert merging instead of outright deletion to see if that further mitigates the multilingual tax.
The takeaway for engineers is clear: If your model needs to serve a specific language, you must calibrate and evaluate it in that language. English is not a universal proxy for "importance."
Summary updated