Introduction
Machine Translation (MT) evaluation has shifted from surface-level metrics like BLEU toward "learned metrics" such as COMET or BLEURT. While these models offer higher semantic accuracy, they suffer from the "Curse of Multilinguality." To support a wide array of languages, these models must be massive, leading to high computational costs and evaluation bottlenecks. This research addresses this challenge by developing compact, high-performance learned metrics that retain the accuracy of large-scale models while significantly reducing their footprint.
Methodology: Overcoming the Curse of Multilinguality
The researchers utilize RemBERT (a rebalanced mBERT) as a foundation and employ three primary strategies to shrink the model without sacrificing performance.
1. Knowledge Distillation
Instead of training small models directly on sparse human-labeled data, the authors use a large "teacher" model (RemBERT-32) to generate "pseudo-labels" for a smaller "student" model. This provides a smoother learning signal and allows the student to learn from a much larger volume of data.
2. Massive Synthetic Data Generation
To facilitate distillation, an 80-million sentence corpus was generated from Wikipedia. Synthetic translation pairs were created using perturbations such as word substitution (via mBERT), back-translation, and word dropping to simulate various levels of MT quality.
3. 1-to-N Distillation
To bypass the capacity bottleneck where one model struggles to learn 100+ languages, the authors trained multiple specialized student models. Each student is focused on a specific cluster of related languages (e.g., Germanic or Romance families), allowing for higher linguistic density in a compact parameter space.
Results and Discussion
The proposed distillation framework demonstrates that model capacity is indeed the primary bottleneck for multilingual transfer. By specializing models and using teacher signals, the researchers achieved significant efficiency gains.
Performance Retention
The primary student model, RemBERT-12 (167M parameters), retained 92.6% of the performance of the massive RemBERT-32 teacher (579M parameters). Furthermore, the distillation method provided up to a 10.5% improvement over standard fine-tuning of the same small architectures.
Computational Efficiency
One of the most impactful outcomes is the reduction in runtime. The distilled models are significantly faster than both the teacher model and other state-of-the-art metrics.
As shown in the runtime analysis, RemBERT-12 offers a 2.5x to 3x speedup compared to the teacher model. This makes high-quality MT evaluation accessible on hardware with more limited resources, including scenarios where GPU acceleration is unavailable.
Conclusion
This work demonstrates that through knowledge distillation and language specialization, it is possible to create MT evaluation metrics that are both accurate and efficient. By moving away from "one-size-fits-all" massive models toward specialized, compact students, researchers can overcome the curse of multilinguality. Future work will explore further compression techniques like pruning and quantization to enable these metrics for edge computing and mobile applications.
Summary updated