Learning Compact Metrics for MT

Amy Pu, Hyung Won Chung, Ankur P. Parikh, Sebastian Gehrmann, Thibault Sellam

Abstract

Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.

Introduction

Evaluating machine translation (MT) is notoriously difficult. While traditional overlap-based metrics like BLEU are fast, they often fail to capture semantic nuances. Modern "learned" metrics like BLEURT and COMET have bridged this gap by leveraging massive, multilingual pre-trained models. However, these models come with a heavy "tax": they are computationally expensive, slow to run, and difficult to deploy alongside the very MT systems they are meant to evaluate.

The current gold standard for MT evaluation involves using large-scale multilingual models (like XLM-RoBERTa or RemBERT) as backbones for regression tasks. These models are excellent at "cross-lingual transfer"—learning from high-resource languages like English to evaluate low-resource ones. However, they face a bottleneck known as the curse of multilinguality: to maintain high accuracy across many languages, the model must be scaled up significantly.

Methodology

The authors utilize RemBERT (a rebalanced version of mBERT) as their foundation and employ three primary strategies to compress it:

  1. Knowledge Distillation: Instead of training a small model from scratch on human ratings (which are scarce), they use a large "teacher" model (RemBERT-32) to generate "pseudo-labels" for a massive amount of unlabeled text. The smaller "student" models (RemBERT-3, 6, and 12) then learn to mimic the teacher.
  2. Synthetic Data Generation: Distillation requires a lot of data. Since they lacked a massive corpus of (Reference, Translation) pairs, they created one. They took 80 million sentences from Wikipedia across 13 languages and applied "perturbations" (word substitution, back-translation, and word dropping) to create synthetic translations.
  3. 1-to-N Distillation: To bypass the capacity bottleneck, they experimented with training multiple specialized student models. Instead of one small model trying to learn 100+ languages, they trained "specialists" on clusters of related languages.
The knowledge distillation pipeline for training compact MT evaluation metrics.

Key Results and Discussion

The experiments confirm that for smaller models, adding more languages during pre-training actually hurts performance on specific tasks as the model's capacity becomes "saturated." Distillation allows students to bypass this "curse" by learning from a teacher that has already mastered multilinguality.

Efficiency and Performance

The RemBERT-12 student (167M parameters) achieved 92.6% of the performance of the RemBERT-32 teacher (579M parameters). Furthermore, the distillation method provided up to a 10.5% improvement over "vanilla" fine-tuning of the same small models.

Runtime comparison between the teacher (RemBERT-32) and distilled student models (RemBERT-12) with GPU enabled and disabled.

On a GPU, the optimized compact models achieved a 2.5–3x speedup over the teacher model. The models also demonstrated strong zero-shot success, performing well even on languages they weren't explicitly fine-tuned on, proving that the cross-lingual transfer capabilities were successfully distilled.

Conclusion

While the results are promising, the authors note that training a massive teacher model is still a prerequisite. Future work may explore other compression techniques like pruning and quantization to shrink the models even further, as well as expanding the linguistic coverage of the synthetic data. Overall, this approach demonstrates that high-quality MT evaluation doesn't always require massive computational resources at inference time.

Summary updated

Abstract

Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.
BibTeX
@inproceedings{pu2021learning,
  author = {Amy Pu and Hyung Won Chung and Ankur P. Parikh and Sebastian Gehrmann and Thibault Sellam},
  title = {{Learning Compact Metrics for MT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=gC_IRZLEvpb},
}
Figure