TL;DR
This paper presents a straightforward yet effective approach to machine translation (MT) quality estimation (QE) that leverages readily available phrase tables from statistical MT systems. By analyzing word and gap alignments, their method achieves competitive results, particularly in identifying where insertions are needed in translated text, and offers a computationally efficient alternative to more complex models.
What Problem Does This Paper Address?
Machine translation (MT) systems are increasingly sophisticated, but ensuring the quality of their output remains a significant challenge. Quality Estimation (QE) aims to predict how good a translation is without relying on a human reference. This paper focuses on a granular level of QE: word-level quality estimation. Specifically, it tackles the problem of identifying not only which words in a machine-translated sentence are "bad" (i.e., need editing) but also whether insertions are needed in the gaps between words. This is crucial for providing actionable feedback to improve translations, going beyond a simple "good" or "bad" sentence score.
Methods
The core of the UAlacant approach lies in its clever use of existing, publicly available statistical MT phrase tables. These tables, derived from large parallel corpora, map source language (SL) phrases to target language (TL) phrases. The authors extract features by:
- Word-Level Analysis: For each word in a translated sentence, they identify matching sub-segments (n-grams) from the SL and TL phrase tables. Features are derived from these matches, including counts of confirming sub-segments, translation frequencies, and evidence from word alignments that suggest a word should be deleted.
- Gap-Level Analysis: Similarly, for the gaps between words, they look for sub-segments that span these gaps. Features are generated based on how frequently sub-segments cover a gap, translation frequencies, and alignment information that suggests insertions are needed.
- Neural Network Architecture: These extracted features are fed into a simple two-hidden-layer feed-forward neural network (NN). This NN is trained to jointly predict whether a word is "BAD" (needs deletion) or a gap is "BAD" (needs insertion). A key aspect is the use of a fixed-length context window around the word/gap being predicted, making the approach akin to a convolutional method and easier to train and parallelize compared to recurrent NNs.
Datasets
The paper utilizes datasets provided for the WMT 2018 shared task on MT Quality Estimation. These datasets cover four language pairs: English-German (EN-DE), German-English (DE-EN), English-Latvian (EN-LV), and English-Czech (EN-CS). For each language pair, both statistical MT (SMT) and neural MT (NMT) system outputs were used. The datasets were split into training, development, and test sets. External bilingual phrase tables were sourced from OPUS for the relevant language pairs.
Key Insights and Ideas
- Leveraging Existing Resources: The paper demonstrates that readily available, albeit "black-box," phrase tables from SMT systems can be a powerful and cost-effective source of information for MT quality estimation, without needing direct access to the MT system's internals.
- Granular Quality Estimation: By focusing on both word deletions and insertion needs, the approach provides a more detailed and useful assessment of translation quality than sentence-level scores alone.
- Simplicity and Efficiency: The use of feed-forward NNs and a fixed context window leads to a simpler architecture that is easier to train and parallelize, requiring less computational power than models employing recurrent NNs.
- Gap Identification is Key: The results highlight that identifying insertion needs (gaps) is a particularly strong aspect of their method, outperforming other systems in this sub-task for several language pairs.
Key Results
The UAlacant system achieved competitive results in the WMT 2018 shared task.
- Gap Labelling: They ranked first in the sub-task of identifying insertions in gaps for three out of the six datasets (DE-EN SMT, EN-LV SMT, and EN-LV NMT).
- Word Labelling: Performance in word labelling was generally poorer, often in the lower part of the rankings.
- Sentence-Level Estimation: Their sentence-level QE submission, which approximated TER using word and gap labels, performed substantially better than the baseline for most datasets and even ranked third for EN-LV NMT.
- Computational Efficiency: Training the system, even with a large feature set, took approximately 2.5 hours on a multi-core CPU, demonstrating its computational efficiency.
Limitations and Future Work
- Limited TER Approximation: The authors acknowledge that their method for approximating TER is limited, as it doesn't fully account for block shifts or the exact number of words to be inserted in a gap.
- Word Labelling Performance: While strong in gap identification, the system's performance in word labelling was less impressive.
- Baseline Feature Dependency: The availability and quality of baseline features (like part-of-speech tags) varied across datasets, impacting performance. For EN-LV, no baseline features were available.
- Future Work: The paper suggests that further improvements could be made by exploring more sophisticated NN architectures or incorporating additional features. The current approach's simplicity, however, makes it a strong candidate for practical applications where computational resources are a concern.
Summary updated