Abstract
Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.
Introduction
Evaluating machine translation (MT) is notoriously difficult. While traditional overlap-based metrics like BLEU are fast, they often fail to capture semantic nuances. Modern "learned" metrics like BLEURT and COMET have bridged this gap by leveraging massive, multilingual pre-trained models. However, these models come with a heavy "tax": they are computationally expensive, slow to run, and difficult to deploy alongside the very MT systems they are meant to evaluate.
The current gold standard for MT evaluation involves using large-scale multilingual models (like XLM-RoBERTa or RemBERT) as backbones for regression tasks. These models are excellent at "cross-lingual transfer"—learning from high-resource languages like English to evaluate low-resource ones. However, they face a bottleneck known as the curse of multilinguality: to maintain high accuracy across many languages, the model must be scaled up significantly.
Methodology
The authors utilize RemBERT (a rebalanced version of mBERT) as their foundation and employ three primary strategies to compress it:
- Knowledge Distillation: Instead of training a small model from scratch on human ratings (which are scarce), they use a large "teacher" model (RemBERT-32) to generate "pseudo-labels" for a massive amount of unlabeled text. The smaller "student" models (RemBERT-3, 6, and 12) then learn to mimic the teacher.
- Synthetic Data Generation: Distillation requires a lot of data. Since they lacked a massive corpus of (Reference, Translation) pairs, they created one. They took 80 million sentences from Wikipedia across 13 languages and applied "perturbations" (word substitution, back-translation, and word dropping) to create synthetic translations.
- 1-to-N Distillation: To bypass the capacity bottleneck, they experimented with training multiple specialized student models. Instead of one small model trying to learn 100+ languages, they trained "specialists" on clusters of related languages.
Key Results and Discussion
The experiments confirm that for smaller models, adding more languages during pre-training actually hurts performance on specific tasks as the model's capacity becomes "saturated." Distillation allows students to bypass this "curse" by learning from a teacher that has already mastered multilinguality.
Efficiency and Performance
The RemBERT-12 student (167M parameters) achieved 92.6% of the performance of the RemBERT-32 teacher (579M parameters). Furthermore, the distillation method provided up to a 10.5% improvement over "vanilla" fine-tuning of the same small models.
On a GPU, the optimized compact models achieved a 2.5–3x speedup over the teacher model. The models also demonstrated strong zero-shot success, performing well even on languages they weren't explicitly fine-tuned on, proving that the cross-lingual transfer capabilities were successfully distilled.
Conclusion
While the results are promising, the authors note that training a massive teacher model is still a prerequisite. Future work may explore other compression techniques like pruning and quantization to shrink the models even further, as well as expanding the linguistic coverage of the synthetic data. Overall, this approach demonstrates that high-quality MT evaluation doesn't always require massive computational resources at inference time.
Summary updated