English Calibration Erodes Lithuanian in Expert-Pruned Mixture-of-Experts Language Models

Tomas Rasymas, Povilas Kvedaras

The promise of Mixture-of-Experts (MoE) models lies in their efficiency: they have massive parameter counts but only activate a small fraction of "experts" for any given token. However, storing these models is still expensive. To solve this, researchers use expert pruning—deleting the experts that seem "unimportant" based on a small calibration dataset.

A new paper, "English Calibration Erodes Lithuanian in Expert-Pruned Mixture-of-Experts Language Models," reveals a major pitfall in this process. While pruning based on English data works fine for English, it can catastrophically "break" the model for lower-resource languages like Lithuanian.

TL;DR

Pruning MoE models using only English calibration data severely degrades performance in lower-resource languages, even when English performance remains stable. By simply including target-language data during the pruning process, researchers can preserve multilingual capabilities with almost no cost to English performance.

What Problem Does This Paper Address?

When we prune an MoE model, we use a "calibration set" to see which experts the model actually uses. Standard practice is to use English data, under the assumption that what’s important for English is important for everything else. Previous studies on high-resource languages (like Chinese) suggested this transfer works well.

The authors test this assumption on Lithuanian, a language with complex morphology and much less web presence than English. They wanted to know if the "English-only" pruning strategy creates a "tax" on lower-resource languages that goes unnoticed if you only evaluate in English.

Methods

The researchers pruned four state-of-the-art MoE models (Qwen3-30B, Qwen3.6-35B, Gemma 4, and GLM-4.7-Flash) using REAP (Router-weighted Expert Activation Pruning). REAP scores experts based on how much the model's "router" trusts them and the magnitude of their output.

They tested three calibration strategies:

  1. English only (EN)
  2. Lithuanian only (LT)
  3. A 50/50 mix (MIX)

They removed either 25% or 50% of the experts in every layer and evaluated the resulting compressed models on reading comprehension, translation, and language modeling.

Datasets

  • Calibration: Wikipedia (encyclopedic text) and Europarl (parallel parliamentary proceedings).
  • Evaluation:
    • Belebele: A multilingual reading comprehension benchmark.
  • FLORES-200: Used for machine translation (English ↔ Lithuanian).
  • Wikipedia: Used to measure "bits per byte" (language modeling accuracy) on held-out articles.

Key Insights and Ideas

  • Asymmetric Transfer: This is the "aha moment." Lithuanian calibration preserves English performance remarkably well, but English calibration destroys Lithuanian performance.
  • Routing Concentration: Why does this happen? The authors found that Lithuanian is "pickier" than English. In the final layers of the network, Lithuanian routing is highly concentrated: 55–72% of its selections go to just eight experts. English routing is much more distributed. English calibration often flags these "Lithuanian-critical" experts as unimportant and deletes them.
  • The "Invisible" Damage: At 25% pruning, a model calibrated on English might lose only 1.7 points in English comprehension but a staggering 21.4 points in Lithuanian. If you only test in English, you’d think the compression was a success, while the model has actually become useless for millions of speakers.
Figure 1: Performance drop across models. English-calibrated models (blue) maintain English accuracy but see Lithuanian accuracy (LT) plummet toward chance levels (dotted line).

Key Results

  • Lithuanian Recovery: Switching to Lithuanian or Mixed calibration removed most of the performance gap. For example, the Lithuanian Belebele loss dropped from ~21 points to just ~3 points.
  • Failure Modes: Models pruned with English-only data didn't just get "worse"—they failed in weird ways. They often failed to stop generating (hitting token limits), switched to English mid-sentence, or hallucinated non-existent Lithuanian word forms.
  • Domain vs. Language: Interestingly, the language of the calibration data mattered far more than the domain (Wikipedia vs. Europarl).

Limitations and Future Work

The study focused specifically on Lithuanian. While the authors suspect these findings apply to other lower-resource languages, they haven't tested them yet. Additionally, the study used a fixed token budget for calibration; because Lithuanian is less token-efficient than English, the Lithuanian calibration sets actually contained less "text" than the English ones, making the recovery results even more impressive.

Future work could explore non-uniform pruning (removing more experts from middle layers where languages overlap more) or expert merging instead of outright deletion to see if that further mitigates the multilingual tax.

The takeaway for engineers is clear: If your model needs to serve a specific language, you must calibrate and evaluate it in that language. English is not a universal proxy for "importance."

Summary updated

Abstract

Expert pruning compresses Mixture-of-Experts (MoE) language models by removing the experts that a calibration set marks as unimportant. Studies of high-resource languages largely find that English calibration transfers well to other languages. We test this for Lithuanian: we pruned four MoE models with REAP, removing 25% or 50% of the experts with English, Lithuanian or mixed calibration, and evaluated both languages on reading comprehension, translation and language modelling. English calibration does not transfer to Lithuanian: at 25% pruning, English comprehension fell by a median of 1.7 points but Lithuanian by 21.4. The transfer is asymmetric. Lithuanian calibration removed most of the Lithuanian loss and never significantly hurt English comprehension, which was on average 2.5 points better than under English calibration; only English language modeling got slightly worse. Routing suggests why: in the last layers, Lithuanian sends 55-72% of its expert selections to just eight experts, against 19-39% for English, and English calibration removes some of these experts. Compressed models should be calibrated, and evaluated, in every language they must serve.
BibTeX
@misc{rasymas2026english,
  author = {Tomas Rasymas and Povilas Kvedaras},
  title = {{English Calibration Erodes Lithuanian in Expert-Pruned Mixture-of-Experts Language Models}},
  year = {2026},
  eprint = {2611.07654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07654},
}
Figure