TL;DR
This paper presents NAIST's system for the IWSLT 2021 English-to-Japanese simultaneous text-to-text translation task. They improved upon a standard "wait-k" neural machine translation model by incorporating sequence-level knowledge distillation to encourage more literal translations and target-side chunk shuffling to better handle Japanese sentence structure, achieving competitive results.
What Problem Does This Paper Address?
Automatic simultaneous translation, aiming to translate speech or text in real-time, is a challenging research area. While neural machine translation (NMT) has shown promise, the English-to-Japanese language pair presents unique difficulties due to significant linguistic differences and data scarcity. The core challenge is to translate accurately and fluently while minimizing latency, which often requires making translation decisions with incomplete input. This paper specifically tackles the text-to-text version of this problem for the English-to-Japanese language pair in the IWSLT 2021 evaluation.
Methods
The primary system is built upon a wait-k neural machine translation (NMT) model. In wait-k, the model waits for k input tokens before starting to generate output, balancing latency and context. To improve translation quality, two main techniques were applied:
- Sequence-Level Knowledge Distillation (SKD): This method uses a pre-trained, "teacher" NMT model to generate pseudo-reference translations for the training data. A "student" simultaneous NMT model is then trained to mimic these pseudo-references. The goal is to encourage more literal and NMT-friendly translations, which are beneficial for simultaneous translation where context is limited.
- Target-Side Chunk Shuffling: Japanese is an agglutinative language where word order can be more flexible than in English. This technique augments the training data by reordering chunks (groups of tokens) within the target Japanese sentences. The motivation is to make the model more robust to variations in output word order and potentially improve fluency.
An additional, experimental approach explored was Incremental Next Constituent Label Prediction (NCLP). This method uses a BERT-based classifier to predict the next syntactic constituent in the input sequence, aiming to provide more structural information to the decoder and potentially improve translation timing and accuracy.
Datasets
- Training Data: The models were initially trained on 17.9 million English-Japanese parallel sentences from the WMT20 news task.
- Fine-tuning Data: The models were then fine-tuned on 223 thousand parallel sentences from the IWSLT 2017 dataset.
- Evaluation Data: The IWSLT 2021 development set was used for hyperparameter tuning and evaluating system performance. The official IWSLT 2021 test set was used for final submission results.
- NCLP Training/Development Data: Penn Treebank 3 and NAIST-NTT TED Talk Treebank were used for training and evaluating the NCLP component.
Key Insights and Ideas
- SKD for Literalness: Sequence-level knowledge distillation proved effective in encouraging more literal translations, which is a desirable trait for simultaneous translation where early output generation is prioritized. This helps mitigate the tendency of NMT models to produce overly fluent but potentially delayed translations.
- Chunk Shuffling's Nuance: Target-side chunk shuffling showed mixed results. While a specific shuffling probability (pr=0.02) yielded the best BLEU score and a shorter output length, other probabilities did not perform as well. This suggests that while the concept is promising for Japanese, careful tuning of the shuffling strategy is crucial.
- NCLP's Potential and Pitfalls: The NCLP approach showed promise in generating more natural and shorter translations in qualitative examples. However, it did not outperform the baseline in quantitative metrics. A key limitation identified was that doubling the input sequence length with constituent labels might dilute the information available to the decoder, forcing more aggressive anticipation with less actual input context.
- Latency-Regime Specific Tuning: The authors found that different latency regimes (low, medium, high) benefited from different configurations, indicating that a one-size-fits-all approach is not optimal for simultaneous translation.
Key Results
- The primary system, combining wait-k with SKD and chunk shuffling, achieved competitive results on the IWSLT 2021 En-Ja development set.
- SKD generally provided improvements, especially in lower-latency settings.
- Chunk shuffling with a specific probability (pr=0.02) improved BLEU scores over the wait-10 baseline.
- The official submissions for the IWSLT 2021 test set, particularly the wait-20 + SKD system (medium latency), achieved comparable BLEU to wait-30 systems but were slightly lower than other teams' submissions.
Limitations and Future Work
- Chunk Shuffling Optimization: The authors acknowledge that the chunk shuffling was a perturbation and requires further investigation to optimize its effectiveness. Linguistically motivated chunk reordering is suggested as a future direction.
- NCLP Integration: The NCLP approach, while showing qualitative promise, did not improve quantitative metrics. The authors suggest exploring more effective ways to integrate NCLP, potentially by addressing the issue of input sequence length doubling or finding alternative ways to represent the predicted constituent labels.
- General Performance Gap: The paper notes that their system's BLEU scores were approximately two points lower than other teams in all latency regimes on the test set, indicating room for further improvement.
Summary updated