TL;DR
This paper introduces a novel adversarial training method to disentangle meaning from language-specific information in sentence embeddings. This approach significantly improves unsupervised Quality Estimation (QE) for machine translation, achieving higher correlations with human evaluations, especially in low-resource language pairs.
What Problem Does This Paper Address?
Estimating the quality of machine translation (MT) without human references (unsupervised QE) is crucial because end-users rarely have access to them. While supervised QE models exist, they require expensive human-annotated data. Unsupervised QE methods often rely on multilingual sentence encoders, but these encoders tend to embed language-specific information more strongly than semantic meaning. This leads to embeddings clustering by language rather than by meaning, hindering accurate similarity estimation across languages. Existing methods like DREAM attempt to disentangle meaning and language embeddings but are complex and lack direct supervision to remove language-specific signals from meaning embeddings.
Methods
The authors propose a method that builds upon the DREAM framework but simplifies it by introducing adversarial training. Their approach uses two multi-layer perceptrons (MLPs): one (MLPM) for extracting meaning embeddings and another (MLPL) for language-specific embeddings. These are trained using a combination of four loss functions:
- Reconstruction Loss ($L_R$): Ensures that the meaning and language embeddings can reconstruct the original sentence embedding.
- Cross Reconstruction Loss ($L_C$): Enforces that meaning embeddings are interchangeable between source and target sentences of a parallel pair.
- Language Embedding Loss ($L_L$): Encourages language embeddings of source and target sentences to be distinct.
- Adversarial Loss ($L_A$): This is the core novelty. An adversarial model (MLPD) is trained to identify the language from the meaning embedding. The adversarial loss ($L_A$) then supervises MLPM to produce meaning embeddings that are unidentifiable by MLPD, effectively removing language-specific information.
This adversarial training eliminates the need for random negative pairs used in DREAM, simplifying the architecture and reducing computational costs.
Datasets
The study uses publicly available bilingual corpora for training, including:
- 1M sentence pairs for high-resource languages (en-de, en-zh).
- 200k sentence pairs for medium-resource languages (ro-en, et-en).
- 50k sentence pairs for low-resource languages (ne-en, si-en).
For evaluation, the WMT20 QE task dataset is used, consisting of 6 language pairs, each with 1k source and machine-translated output sentence pairs manually labeled with quality scores.
Key Insights and Ideas
- Adversarial training is a powerful tool for disentanglement: By training a classifier to detect language-specific information and then forcing the main model to fool this classifier, the authors effectively remove unwanted signals from the meaning embeddings.
- Simplicity can be an advantage: Eliminating the need for negative sampling (as in DREAM) leads to a simpler model and more efficient training.
- Language embedding loss is crucial: The ablation study highlights that while adversarial loss ($L_A$) is important, the language embedding loss ($L_L$) plays a critical role in ensuring the separation of meaning and language information.
- Unsupervised QE benefits significantly from disentangled embeddings: The improved performance, especially on low-resource languages, demonstrates the value of this approach for practical MT quality assessment.
Key Results
The proposed method, referred to as "Ours," consistently outperformed the baseline LaBSE (a state-of-the-art multilingual sentence encoder) and DREAM across various language pairs. Notably, it achieved the highest Pearson correlation coefficients with human scores on average for all language pairs, particularly excelling in low-resource settings. For instance, on the 'ne-en' low-resource pair, the proposed method achieved a correlation of 0.634, significantly higher than LaBSE (0.545) and DREAM (0.636). The method also outperformed other unsupervised QE methods like LASER and mSBERT, and even surpassed the supervised Predictor-Estimator model on average.
Limitations and Future Work
The authors acknowledge that while their method outperforms DREAM, the language embedding loss ($L_L$) and adversarial loss ($L_A$) are the most critical components for performance. They suggest future work could explore using language-specific embeddings for assessing sentence fluency and extending the disentanglement approach to styles and meanings for style-transfer research. The paper also notes that while D-TP and Prism achieved higher scores in high-resource pairs, their methods have limitations (e.g., requiring access to MT model parameters or large bilingual corpora) that restrict their applicability, especially in low-resource scenarios where the proposed method shines.
Summary updated