Abstract: Shuo Sun, Ahmed El-Kishky, Vishrav Chaudhary, James Cross, Lucia Specia, Francisco Guzmán. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction
Machine translation Quality Estimation (QE) is the task of predicting translation quality without a human-written reference. While state-of-the-art models based on XLM-RoBERTa have achieved high accuracy, they are often too computationally expensive for production environments. A typical "large" QE model can take 370ms per sentence and require over 1GB of memory, making it impractical for high-volume applications like social media feeds.
This paper addresses the mismatch between evaluation metrics (continuous regression scores) and real-world usage (binary "good enough" filters). The authors argue that reframing QE as a classification task allows for much more aggressive model compression without sacrificing utility.
Methodology
The study explores three primary strategies for improving QE efficiency:
- Task Reframing: Shifting from continuous regression (0-100) to binary classification. Two thresholds were established based on human guidelines: $\ge 51$ (understandable) and $\ge 70$ (semantically accurate).
- Model Compression: Three techniques were evaluated:
- Layer Pruning: Removing the top $N$ encoder layers.
- Token Pruning: Removing redundant word vectors during inference.
- Module Replacement: Using knowledge distillation to replace $N$ teacher layers with a single student layer.
- Bilingual vs. Multilingual Models: Comparing specialized models against universal multilingual models under compression.
Experiments were conducted on the MLQE-PE dataset (WMT 2020), covering seven language pairs across high, medium, and low-resource levels.
Key Results
The central finding is that classification-based models are significantly more robust to compression than regression-based ones.
As shown in the regression results above, stripping layers leads to a sharp decline in Pearson correlation. At a 24x speedup, regression performance drops by approximately 34%.
In contrast, the classification model remains remarkably stable. At the same 24x speedup, classification performance (F1) only degrades by about 11%. Simple layer pruning emerged as the most effective compression method, achieving an 8x speedup while retaining 94.6% of original performance.
Discussion and Insights
- Robustness of Classification: Regression is "brittle" because it attempts to capture fine-grained nuances that are lost during compression. Classification focuses on the decision boundary, which is more resilient.
- The Multilingual Tax: Multilingual models require more parameters to maintain generalization across languages. Consequently, they degrade faster than bilingual models at high compression rates (e.g., dropping >12 layers).
- Model Size: The performance gap between "Large" and "Base" models is much smaller in classification tasks (1-7%) than in regression (11-56%), suggesting that smaller base models are often sufficient for practical classification-based QE.
Conclusion
Reframing Quality Estimation as a classification task enables the deployment of highly efficient models in production. By prioritizing the binary "acceptability" of a translation over a precise score, developers can achieve up to 24x speedups with minimal loss in practical utility. Future work should focus on improving the compression of multilingual models to maintain their cross-lingual capabilities at smaller scales.
Summary updated