TL;DR
This paper introduces BIT-Xiaomi's system for the AutoSimTrans 2022 simultaneous translation challenge, focusing on improving translation quality and reducing latency. They propose a novel "streaming chunking" method that integrates with a wait-k policy to better handle source sentence segmentation, leading to significant BLEU score improvements, especially in low-latency scenarios.
What Problem Does This Paper Address?
Simultaneous translation, the real-time conversion of speech or text from one language to another, faces a fundamental trade-off: quality versus latency. To achieve low latency, systems must start translating before the source sentence is complete, which can lead to errors due to incomplete information. Traditional machine translation models trained on full sentences often perform poorly in this prefix-to-prefix inference setting. Existing approaches include fixed policies (like wait-k, which waits for a fixed number of source tokens) and adaptive policies (which use an agent to decide when to translate). However, these methods can still struggle with accurately segmenting source language input, particularly in languages like Chinese where word boundaries aren't always clearly demarcated by spaces, leading to potential misunderstandings and incorrect translations.
Methods
The BIT-Xiaomi system builds upon a standard transformer architecture and incorporates several key techniques:
- Wait-k Policy: This is a core strategy for simultaneous translation, where the system waits for a specified number of source tokens (k) before producing an output. The paper uses a multi-path wait-k approach during pre-training and a simpler wait-k during fine-tuning to enhance performance.
- Streaming Chunking: This is the paper's main novelty. It introduces a streaming segmentation model, based on Chinese RoBERTa, that detects word boundaries in the source stream on-the-fly. This model is trained to predict if a source prefix ends with a complete word. The wait-k policy is then only activated at these detected word boundaries, preventing translation of incomplete word pieces.
- Data Augmentation: To combat limited training data, the authors employ back-translation (BT) and front-translation (FT) to generate synthetic parallel data. They also incorporate character-level augmentations (insertion, deletion, duplication, homophone substitution) for fine-tuning.
- R-Drop Training: This regularization technique is used to improve translation quality by encouraging the model to produce consistent outputs under different dropout masks.
- Data Filtering and Selection: Standard techniques like filtering sentences with few linguistic words, using fast_align for alignment scores, and language identification are applied. Additionally, a language-model-based data selection method is used to prioritize domain-specific data.
Datasets
The system was trained and evaluated on several datasets for different tracks:
- Zh-En Text-to-Text: BSTC (38K sentence pairs), CWMT (9M sentence pairs).
- En-Es Text-to-Text: UN Parallel Corpus (22M sentence pairs).
- Zh-En Audio-to-Text (ASR): BSTC and Allshell (150 hours) for ASR training.
The paper does not explicitly mention the creation of new benchmarks but rather participation in the AutoSimTrans 2022 challenge, which implies existing evaluation protocols.
Key Insights and Ideas
- The Importance of Source Segmentation: The paper strongly demonstrates that accurately segmenting the source stream, especially for languages without clear word delimiters, is crucial for improving simultaneous translation quality. Their streaming chunking method directly addresses this by preventing premature translation of incomplete words.
- Synergy of Wait-k and Chunking: Combining a wait-k policy with their streaming chunking method offers a powerful approach. The chunking identifies natural points to pause and wait, while wait-k controls the overall latency, leading to a better balance.
- Data Augmentation Pays Off: The extensive data augmentation strategies, including BT, FT, and character-level methods, significantly boosted performance, highlighting their importance when dealing with domain-specific or limited data.
- R-Drop for Robustness: The R-Drop regularization proved effective in improving translation quality, suggesting its utility in making models more robust.
Key Results
- Zh-En Text-to-Text: The proposed streaming chunking method, when combined with a wait-k baseline, improved performance by approximately 2 BLEU points in the low-latency regime. Their best wait-k model (M2_wait1-9_wait5) outperformed the organizer's baseline by up to 8 BLEU points.
- Zh-En Audio-to-Text: While the ASR transcription quality was good (0.38 WER on development), the simultaneous translation BLEU score dropped by about 10 BLEU points compared to the text-to-text track, indicating that ASR errors can significantly impact translation quality.
- En-Es Text-to-Text: Due to a lack of speech corpus, data selection and augmentation were not performed. Results are presented in Figure 4, showing performance across different average lagging values.
Limitations and Future Work
- ASR Error Impact: The paper acknowledges that ASR errors in the audio-to-text track negatively affect translation quality.
- Future Work: The authors plan to explore better streaming ASR models and investigate more advanced simultaneous policies to further enhance both latency and quality. They also suggest that their streaming chunking method can be integrated with streaming decoding to improve translation quality.
Summary updated