TL;DR
This paper introduces "Minimalist Quality Estimation" (MQE), a novel approach for assessing machine translation quality using only two simple features: lexical overlap and language model cross-entropy. The authors demonstrate that both supervised (sMQE) and unsupervised (uMQE) variants of MQE achieve competitive results, even outperforming more complex baselines, making it a highly portable and efficient solution for quality estimation across many language pairs and domains.
What Problem Does This Paper Address?
Estimating the quality of machine translation (MT) output without human references is crucial, especially when reference translations are unavailable or costly to produce. Traditional approaches often rely on complex supervised models with a large number of features, requiring significant computational resources and annotated data. This becomes a bottleneck for practical deployment across the vast number of language pairs and domains that exist. Furthermore, state-of-the-art neural network-based methods demand large parallel corpora, which are scarce for many languages. This paper aims to address this by developing minimalist quality estimation systems that are easy to deploy, require minimal resources, and can function effectively even without annotated training data.
Methods
The core of Vicomtech's approach, MQE, is built upon two fundamental features:
- Accuracy (Lexical Translation Overlap): This feature measures how much of the source information is represented in the translation. It's calculated using a modified Jaccard similarity on token multisets, incorporating k-best lexical translations (derived from symmetrized IBM2 word alignment models) and expanding these sets to handle morphological variation and named entities. A "NULL token" is also added to account for covert translations.
- Fluency (Language Model Cross-Entropy): This feature assesses the grammatical correctness of the translated sentence. It's measured by the inverse of the per-word cross-entropy calculated using an n-gram language model. A lower cross-entropy (higher fluency score) indicates a more fluent translation.
Two variants of MQE are proposed:
- uMQE (Unsupervised): This variant combines the accuracy and fluency scores by simply averaging them after min/max feature rescaling. It's designed for scenarios where no annotated data is available for training.
- sMQE (Supervised): This variant uses a Support Vector Regressor (SVR) with a Radial Basis Function kernel, trained on the accuracy and fluency features using available annotated datasets (HTER scores).
Datasets
The experiments were conducted on the WMT 2018 shared task datasets for sentence-level quality estimation. For training the IBM2 and language models, the authors utilized various corpora relevant to the WMT shared tasks for specific domains and language pairs, including:
- WMT 2016 IT translation task data
- WMT 2017 QE task data
- WMT 2018 PE task data
- Technical manuals from OpenOffice and KDE4 (via OPUS repository)
- UFAL medical corpus
- EMEA corpus
- CzEng17 dataset (train-techdoc section)
The paper notes that for the unsupervised uMQE variant, they intentionally ignored task-related annotated data to test it under its intended unsupervised conditions.
Key Insights and Ideas
- Simplicity Trumps Complexity: The paper strongly suggests that minimalist approaches, using only a few well-chosen features, can be highly effective for quality estimation, often outperforming more complex, feature-rich systems.
- Unsupervised QE is Viable: The uMQE variant demonstrates that reliable quality estimation can be achieved without any annotated data, a significant advantage for broad applicability.
- Portability and Efficiency: MQE's reliance on basic features and lack of external tools (like POS taggers or parsers) makes it exceptionally portable and efficient to deploy across diverse languages and domains.
- Data Sparseness Impact: The authors identify data sparseness, particularly in languages with complex morphology, as a factor that can negatively impact the performance of the n-gram-based fluency feature.
Key Results
The results on the WMT 2018 test sets were generally satisfactory.
- Both sMQE and uMQE variants consistently outperformed supervised baselines.
- In several language pairs (e.g., English-Latvian), the systems ranked among the top five best systems.
- For English-Latvian, sMQE and uMQE ranked second and third, respectively.
- For German-English, the variants ranked fourth and fifth.
- The performance was relatively weaker for English-Czech and English-German, which the authors attribute to data sparseness issues affecting the fluency feature.
Limitations and Future Work
- Data Sparseness: The authors acknowledge that data sparseness, especially in languages with complex morphology, can limit the effectiveness of the n-gram-based fluency feature.
- Future Work:
- Investigate the use of continuous space language models to better handle data sparseness issues in complex morphology languages.
- Explore variants of the accuracy measure to further refine the approach.
- Evaluate the proposed minimalist approach in more detail across various aspects of quality estimation.
Summary updated