TL;DR
This paper presents the USST system for simultaneous Chinese-English translation, which achieved second place in the AutoSimTrans 2022 challenge. The system leverages a deep Transformer architecture, data synthesis techniques, and a novel "in-domain mixed fine-tuning" approach to bridge the gap between general and spoken domains, ultimately improving translation quality and reducing latency.
What Problem Does This Paper Address?
Simultaneous Machine Translation (SMT) aims to translate speech in real-time, meaning the translation must be generated before the speaker finishes their utterance. This presents a fundamental trade-off: to achieve high translation quality, the system needs to read more of the source sentence, which increases latency. Conversely, to minimize latency, the system must start translating with less source information, potentially sacrificing quality. Existing approaches often fall into fixed "wait-k" policies (waiting a predetermined number of words) or more complex adaptive policies. This paper tackles the challenge of improving SMT quality, particularly for spoken language domains, by developing a system that balances translation accuracy with low latency.
Methods
The USST system builds upon a Transformer architecture with a deep encoder, enhanced by DeepNet's techniques for stable training of deeper models.
- Pre-training: The model is initially pre-trained on a large general corpus (CWMT21).
- Data Synthesis: To augment the training data, the authors employ self-training and back-translation, generating an additional 5.7 million pseudo-parallel sentences.
- Domain Adaptation: A key contribution is the "in-domain mixed fine-tuning" approach. This method addresses the common issue of overfitting when fine-tuning on limited spoken domain data. Instead of simply mixing general and spoken corpora, they filter the general corpus to select sentences most similar to the spoken domain, creating a more targeted mixed dataset. This avoids the need for domain tags, which are problematic for prefix-to-prefix SMT models.
- Wait-k Policy: The full-sentence translation model is converted into a simultaneous translation model using a fixed wait-k policy. This means the system reads
k-1source subwords, then predicts a target subword, alternating until the end of the sentence.
Datasets
- CWMT21: A large general-domain Chinese-English parallel corpus (9.1 million sentence pairs) used for pre-training.
- BSTC: The Baidu Speech Translation Corpus (37K sentence pairs), a spoken-domain corpus used for fine-tuning.
- CWMT Monolingual Data: 10 million Chinese monolingual sentences from CWMT were used for synthetic data generation.
Key Insights and Ideas
- Bridging the Domain Gap: Directly fine-tuning on spoken domain data can lead to overfitting. The proposed "in-domain mixed fine-tuning" effectively mitigates this by carefully selecting in-domain data from a larger general corpus, creating a more robust adaptation strategy.
- No Domain Tags Needed: For simultaneous translation, appending domain tags to the source sentence is not feasible as the model only sees prefixes. The in-domain mixed fine-tuning approach circumvents this by creating a mixed dataset that inherently represents the spoken domain.
- Deep Transformers for SMT: Employing a deep Transformer encoder, as in previous work, is shown to be beneficial for capturing rich source representations, crucial for translation quality.
- Ensemble Benefits: Averaging checkpoints from in-domain mixed fine-tuning further improved BLEU scores, demonstrating the effectiveness of ensemble methods.
Key Results
The USST system achieved second place in the Chinese-English streaming translation task of AutoSimTrans 2022.
- The "in-domain mixed fine-tuning" method resulted in a 0.6 BLEU improvement over standard mixed fine-tuning.
- Ensembling two checkpoints of the in-domain mixed fine-tuning model further boosted performance by 0.3 BLEU points, reaching 21.75 BLEU on the BSTC development set, which is 1.3 BLEU higher than simple fine-tuning.
- The system demonstrated a good balance between translation quality (BLEU) and latency (Average Lagging), as visualized in Figure 2.
Limitations and Future Work
- Perplexity Difference: While the paper explored using perplexity difference for in-domain data selection, it proved less effective than the absolute value of perplexity difference.
- Future Work: The authors plan to:
- Validate their in-domain mixed fine-tuning approach on more datasets.
- Investigate novel domain adaptation methods.
- Research dynamic read-write policies to further optimize the quality-latency trade-off in simultaneous translation.
Summary updated