Abstract: Sentence-level Quality estimation (QE) of machine translation is traditionally formulated as a regression task, and the performance of QE models is typically measured by Pearson correlation with human labels. Recent QE models have achieved previously-unseen levels of correlation with human judgments, but they rely on large multilingual contextualized language models that are computationally expensive and make them infeasible for real-world applications. In this work, we evaluate several model compression techniques for QE and find that, despite their popularity in other NLP tasks, they lead to poor performance in this regression setting. We observe that a full model parameterization is required to achieve SoTA results in a regression task. However, we argue that the level of expressiveness of a model in a continuous range is unnecessary given the downstream applications of QE, and show that reframing QE as a classification problem and evaluating QE models using classification metrics would better reflect their actual performance in real-world applications.
Introduction
Quality Estimation (QE) is often referred to as the "holy grail" of machine translation (MT). It aims to predict the quality of a translation at runtime without access to a reference sentence. While recent advances in multilingual transformers like XLM-R have pushed QE performance to new heights, these models come with a significant computational burden. In production settings, such as social media platforms or translation agencies, a QE model that is slower than the MT engine itself is a major bottleneck.
The Regression Problem
The standard approach in the QE community is to treat quality prediction as a regression task, typically aiming to match human-annotated Direct Assessment (DA) scores on a scale of 0-100. Performance is then measured using Pearson correlation.
The authors of this paper identify a "regression bottleneck": to maintain high correlation across a continuous range, models require a high degree of representational expressiveness. When these models are compressed using standard techniques like layer pruning or knowledge distillation, their ability to provide precise continuous scores "falls apart" much faster than their performance on other NLP tasks.
Methodology: Compression and Reframing
To address the efficiency gap, the authors evaluate three primary compression strategies:
- Layer Pruning: Systematically removing the top $N$ encoder layers.
- Token Pruning: Removing redundant word vectors based on attention scores to reduce the sequence length processed by deeper layers.
- Module Replacement: A form of knowledge distillation where multiple teacher layers are compressed into a single student layer.
The key innovation, however, is the reframing of the task. The authors argue that for most downstream applications, the exact score (e.g., 65 vs 67) is irrelevant. What matters is a binary decision: is the translation "Acceptable" or "Not Acceptable"? By converting DA scores into binary labels (using thresholds like $\ge 51$ for understandability and $\ge 70$ for high quality), they test the robustness of compressed models in a classification setting.
Results and Discussion
The experiments were conducted on the MLQE-PE dataset, covering a diverse range of language pairs. The findings reveal a significant disparity in how compression affects regression versus classification.
As illustrated in Figure 3, regression performance (Pearson $\rho$) degrades sharply as speedup increases. In contrast, Figure 4 shows that classification performance (F1 score) is remarkably robust.
Specifically, layer pruning allowed for a 24x speedup while retaining 88.8% of the classification F1 score. At a 2x speedup, while regression performance dropped by over 9%, classification performance remained nearly identical to the uncompressed baseline. This suggests that while compressed models may lose the fine-grained precision needed for regression, they retain the coarse-grained information necessary for accurate binary decisions.
The Multilingual Capacity Limit
The study also highlights a trade-off between multilinguality and compressibility. While large multilingual models are excellent for zero-shot tasks, they appear to have a "capacity limit" when compressed. At high levels of pruning (dropping more than 12 layers), bilingual models—trained specifically for one language pair—consistently outperformed multilingual models. This suggests that for extremely small, high-speed deployments, specialized bilingual models may be preferable.
Conclusion
This research demonstrates that the perceived "failure" of model compression in QE is largely an artifact of the regression-based evaluation framework. By shifting the focus to classification, we can unlock massive efficiency gains (up to 24x) with minimal impact on real-world utility. Future research should look toward more advanced distillation techniques to see if the gap between small and large models can be narrowed even further for more demanding thresholds.
Summary updated