TL;DR
This paper tackles the "restricted translation" task, where machine translation systems must include specific target words or phrases. They adapt a semi-autoregressive model, RecoverSAT, to handle this by forcing the inclusion of these "restricted target vocabularies" (RTVs). Crucially, they demonstrate that ordering these RTVs based on their alignment with source words significantly improves translation quality.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) models have achieved remarkable fluency, but often struggle with strict constraints. The "restricted translation" task, as defined by the WAT 2021 challenge, presents a specific challenge: generating a translation that must contain a given set of target words or phrases (RTVs). While previous work has explored terminological constraints, it often relied on bilingual dictionaries or focused solely on autoregressive NMT models. This paper addresses the gap of incorporating target-side constraints into semi-autoregressive NMT systems, which offer a balance between speed and accuracy. A key difficulty is determining the optimal order in which to insert these RTVs, as simply inserting them randomly can lead to suboptimal translations.
Methods
The authors employ a semi-autoregressive model called RecoverSAT (Ran et al., 2020). Unlike fully autoregressive models that generate tokens one by one, RecoverSAT generates sentences in segments, with tokens within each segment generated autoregressively. This allows for faster decoding than purely autoregressive models while mitigating some of the multi-modality issues of non-autoregressive models.
To handle the restricted translation task, they adapt RecoverSAT's "forced translation" capability. Instead of forcing a single word, they aim to insert multiple RTVs. A critical aspect of their approach is sorting the RTVs based on source-side alignment. They use GIZA++ to align each RTV with words in the source sentence. The RTVs are then inserted in the order corresponding to their aligned source words. If an RTV is a phrase, they find the source word with the highest alignment score for that phrase.
For baseline and pre-training, they utilize the Transformer (base) model. They also refine their training data using a forward-translation approach, inspired by previous work, to improve the quality of synthetic data.
Datasets
- ASPEC dataset: Used for Japanese-to-English translation. It comprises 3M sentences for training, 1,790 for validation, and 1,812 for testing. The training data was refined using forward-translation.
- WAT 2021 restricted translation task: The specific task setting for evaluation.
Key Insights and Ideas
- Order Matters: The most significant insight is that the order in which RTVs are inserted into the translation has a substantial impact on the final BLEU score. Random ordering is significantly worse than ordering based on source-side alignment.
- Source Alignment as a Heuristic: Using GIZA++ to align RTVs with source words provides an effective heuristic for determining the insertion order, leveraging the inherent structure of the source sentence.
- RecoverSAT's Adaptability: The semi-autoregressive nature of RecoverSAT, particularly its segment-based generation and "forced translation" capability, makes it a suitable candidate for handling multiple target-side constraints.
- Trade-off with Number of Segments: Increasing the number of segments in RecoverSAT can improve the ability to insert all RTVs but may lead to a decrease in overall translation accuracy, pushing the model closer to a non-autoregressive behavior.
Key Results
- Improved BLEU Score: Sorting RTVs using GIZA++ significantly improved the BLEU score compared to random ordering.
- High Consistency Score: The proposed method achieved a high consistency score (over 96%) for inserting all RTVs into the output sentences.
- Official Evaluation: The RecoverSAT model with 14 segments and sorted RTVs achieved a BLEU score of 25.29, RIBES of 0.653597, and AMFM of 0.612290.
- Comparison with Oracle Order: While sorted order improved results, the "oracle order" (using the reference translation's RTV order) yielded the highest BLEU score (31.14), highlighting that perfect source-target alignment is still an open challenge.
- Consistency vs. BLEU: The paper shows that while forced translation with RTVs dramatically increases the consistency score (approaching 1.000), the BLEU score can sometimes be lower than the baseline Transformer without RTVs, especially when RTVs are inserted without proper ordering.
Limitations and Future Work
- Determining Optimal Order: The authors acknowledge that while source-side alignment improved results, there is still room for improvement in determining the best order for inserting RTVs. Future work should investigate more sophisticated methods for this.
- Language Pair Differences: The difference in word order between Japanese and English is cited as a reason why the sorted order still lags behind the oracle order.
- DEL Token Issue: In some cases, RecoverSAT might generate a "DEL" token, effectively deleting segments that begin with RTVs, leading to missed constraints even with sufficient segments.
- Segment Length and Number: The paper explores varying the number of segments, noting a trade-off between the ability to insert all RTVs and overall translation accuracy. Further research could optimize this balance.
Summary updated