TL;DR
This paper tackles the challenge of neural machine translation (NMT) with strict lexical constraints, where specific words must appear in the translated output. The authors propose novel methods for automatically extracting these constraints from training data, focusing on proper nouns and mistranslated words, which significantly improve translation quality and consistency compared to random constraint selection.
What Problem Does This Paper Address?
Traditional neural machine translation (NMT) systems excel at generating fluent and contextually relevant translations. However, certain applications, particularly in technical or legal domains, require precise adherence to specific terminology, including technical terms and proper nouns. The WAT 2021 restricted machine translation task specifically addresses this by providing a list of required target words for each source sentence. The core challenge lies in effectively incorporating these constraints into the NMT process, especially during training when these explicit constraints are not provided. Simply forcing the NMT to include these words can lead to unnatural translations or even break the model's fluency.
Methods
The paper builds upon existing lexically-constrained NMT methods, which typically involve concatenating the source sentence and the constrained words with a special token before feeding them into the NMT encoder. The novelty of this work lies in its sophisticated methods for automatically extracting these constraints from the target sentences in the training data.
The authors propose two main constraint extraction strategies:
- Proper-Noun Constraint: This method leverages named-entity recognition (NER) tools (GiNZA for Japanese, spaCy for English) to identify proper nouns in the target sentences. The rationale is that important technical terms and proper nouns are often critical for accurate translation.
- Mistranslated-Word Constraint: This strategy aims to capture words that a standard NMT model might struggle with. They train a preliminary NMT model, translate the training source sentences, and then identify words present in the target sentence but missing from the preliminary translation. This captures words that are either mistranslated or under-translated.
These two methods can be used independently or combined ("Prop. & Mistrans.") to create a more comprehensive set of constraints.
Datasets
The primary dataset used is the ASPEC-JE (Asian scientific paper excerpt corpus), a Japanese-English dataset. The authors utilize the first 2.0 million bilingual pairs from this dataset for training, excluding the remaining 1.0 million pairs due to potential noise. The dev, devtest, and test sets of ASPEC-JE are used for evaluation, and these sets are provided with target vocabulary lists (constraints) by the WAT 2021 organizers. No other external resources were used for training.
Key Insights and Ideas
- Constraint Quality Over Quantity: The results strongly suggest that how constraints are extracted is more important than simply having a large number of them. The "Proper-noun" method, despite extracting fewer words on average than the "Random-word" method, yielded significantly better BLEU scores.
- Proper Nouns and Mistranslations are Key: The proposed methods, focusing on proper nouns and words likely to be mistranslated, significantly outperform a baseline that randomly selects constraints. This highlights the importance of identifying semantically crucial or problematic words.
- Combined Approach is Best: The "Prop. & Mistrans." method, which combines both proper-noun and mistranslated-word constraints, consistently achieved the best results across various metrics, demonstrating the synergistic benefit of these two extraction strategies.
- Impact on Consistency and Accuracy: The proposed methods not only improve translation accuracy (BLEU scores) but also dramatically increase the consistency rate, meaning more translations successfully incorporate the required words.
Key Results
The experimental results, summarized in Table 2 and Table 3, show significant improvements over the baseline and random-word constraint methods:
- Japanese English Task:
- The "Prop. & Mistrans." method achieved a consistency rate of 96.0% and a BLEU score of 43.9, a substantial improvement over the baseline (52.3% consistency, 29.3 BLEU) and random-word (78.6% consistency, 29.5 BLEU).
- Human evaluation scores (DA and CA) also showed a significant boost for the "Prop. & Mistrans." method.
- English Japanese Task:
- The "Prop. & Mistrans." method achieved a consistency rate of 97.4% and a BLEU score of 53.2, outperforming the baseline (61.7% consistency, 45.9 BLEU) and random-word (77.8% consistency, 37.4 BLEU).
- Again, human evaluation scores were notably higher.
Limitations and Future Work
- Limitations:
- The paper notes that the "Proper-noun" constraint alone only accounted for 21% of the Japanese dev data constraints, indicating its limitations in covering all necessary terms.
- While the "Prop. & Mistrans." method improved consistency, it still didn't guarantee 100% constraint satisfaction for all words in the output. The authors introduced a post-processing "rule" to handle unsatisfied words in the official evaluation, which is a workaround rather than an integrated solution.
- Future Work:
- The authors plan to integrate their proposed constraint extraction methods with NMT using a grid beam search decoding algorithm. This is a promising direction to further improve constraint satisfaction and potentially achieve even better translation quality by exploring a wider decoding space.
Summary updated