Learning Compact Metrics for MT

Amy Pu, Hyung Won Chung, Ankur P. Parikh, Sebastian Gehrmann, Thibault Sellam

Introduction

Machine Translation (MT) evaluation has shifted from surface-level metrics like BLEU toward "learned metrics" such as COMET or BLEURT. While these models offer higher semantic accuracy, they suffer from the "Curse of Multilinguality." To support a wide array of languages, these models must be massive, leading to high computational costs and evaluation bottlenecks. This research addresses this challenge by developing compact, high-performance learned metrics that retain the accuracy of large-scale models while significantly reducing their footprint.

Methodology: Overcoming the Curse of Multilinguality

The researchers utilize RemBERT (a rebalanced mBERT) as a foundation and employ three primary strategies to shrink the model without sacrificing performance.

Overview of the teacher-student distillation pipeline.

1. Knowledge Distillation

Instead of training small models directly on sparse human-labeled data, the authors use a large "teacher" model (RemBERT-32) to generate "pseudo-labels" for a smaller "student" model. This provides a smoother learning signal and allows the student to learn from a much larger volume of data.

2. Massive Synthetic Data Generation

To facilitate distillation, an 80-million sentence corpus was generated from Wikipedia. Synthetic translation pairs were created using perturbations such as word substitution (via mBERT), back-translation, and word dropping to simulate various levels of MT quality.

3. 1-to-N Distillation

To bypass the capacity bottleneck where one model struggles to learn 100+ languages, the authors trained multiple specialized student models. Each student is focused on a specific cluster of related languages (e.g., Germanic or Romance families), allowing for higher linguistic density in a compact parameter space.

Results and Discussion

The proposed distillation framework demonstrates that model capacity is indeed the primary bottleneck for multilingual transfer. By specializing models and using teacher signals, the researchers achieved significant efficiency gains.

Performance Retention

The primary student model, RemBERT-12 (167M parameters), retained 92.6% of the performance of the massive RemBERT-32 teacher (579M parameters). Furthermore, the distillation method provided up to a 10.5% improvement over standard fine-tuning of the same small architectures.

Computational Efficiency

One of the most impactful outcomes is the reduction in runtime. The distilled models are significantly faster than both the teacher model and other state-of-the-art metrics.

Runtime comparison between teacher and student models with and without GPU acceleration.

As shown in the runtime analysis, RemBERT-12 offers a 2.5x to 3x speedup compared to the teacher model. This makes high-quality MT evaluation accessible on hardware with more limited resources, including scenarios where GPU acceleration is unavailable.

Conclusion

This work demonstrates that through knowledge distillation and language specialization, it is possible to create MT evaluation metrics that are both accurate and efficient. By moving away from "one-size-fits-all" massive models toward specialized, compact students, researchers can overcome the curse of multilinguality. Future work will explore further compression techniques like pruning and quantization to enable these metrics for edge computing and mobile applications.

Summary updated

Abstract

Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.
BibTeX
@inproceedings{pu2021learning,
  author = {Amy Pu and Hyung Won Chung and Ankur P. Parikh and Sebastian Gehrmann and Thibault Sellam},
  title = {{Learning Compact Metrics for MT}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=gC_IRZLEvpb},
}
Figure