TL;DR
Xiaomi's team tackled the IWSLT 2020 Chinese-Japanese open domain translation task by employing a comprehensive suite of advanced NMT techniques, including diverse Transformer architectures, extensive data filtering and augmentation, and sophisticated ensembling and reranking methods. Their primary Chinese-Japanese system achieved a remarkable second place in character-level BLEU, demonstrating the effectiveness of their integrated approach.
What Problem Does This Paper Address?
The paper addresses the challenge of building high-performing Neural Machine Translation (NMT) systems for the Chinese-Japanese language pair within the context of the IWSLT 2020 open domain translation task. Open domain translation is particularly difficult because it requires systems to generalize well to unseen topics and domains, unlike specialized systems trained on narrow, in-domain data. The quality and quantity of training data are critical for NMT performance, and real-world data, especially from web crawls, often contains significant noise and errors that can degrade translation accuracy. Therefore, a key problem is how to effectively filter noisy data, leverage available data, and build robust translation models that can achieve competitive results in a challenging open-domain setting.
Methods
The authors employed a multi-faceted approach to enhance their NMT systems:
- Diverse Architectures: Beyond the standard Transformer-Big model, they explored several variants to increase model diversity. These included:
- Wider models: Increasing the dimension of feed-forward layers.
- Deeper models: Stacking more encoder and decoder layers (e.g., Transformer-Deep-12-12).
- DLCL models: Transformer models with dynamic linear combinations of layers to better memorize previous layer outputs.
- NAS-based models: Utilizing the computationally efficient Transformer-Evolved model discovered through Neural Architecture Search (NAS).
- Data Filtering and Augmentation: Recognizing the impact of noisy data, they implemented a two-stage filtering process:
- Rule-based Filtering: Applying heuristics to remove illegal sentence pairs based on length, character ratios, and format inconsistencies.
- Model-based Scoring: Using translation models (BEER metric), SBERT embeddings (cosine similarity), word alignment models (fast_align probability), and N-gram Language Models (perplexity) to score and prune sentence pairs.
- Data Diversification: Employing back-translation (BT) and knowledge distillation (KD) to generate synthetic parallel data and augment the training set. They also explored right-to-left (R2L) NMT models for data augmentation.
- Model Ensembling and Reranking:
- Ensembling: Combining the predictions of the top 4 best-performing single models on the development set to improve robustness.
- Reranking: Using an ensemble of models (S2T-L2R, S2T-R2L, T2S-L2R) to generate diverse translation hypotheses, then reranking them using features like neural LM perplexity, SBERT similarity, and pairwise BLEU distance, trained with RankSVM.
Datasets
The primary datasets used were:
- IWSLT 2020 Shared Task Data: Provided parallel data for Chinese-Japanese and Japanese-Chinese.
- Large-scale but noisy parallel data: Extracted from web crawls, requiring extensive filtering.
- External monolingual data: Used for back-translation and language model training.
The paper details the process of filtering and augmenting these datasets, resulting in training data sizes ranging from 12M to 84M sentence pairs.
Key Insights and Ideas
- Data Quality is Paramount: The paper strongly emphasizes that effective data filtering is crucial for NMT performance, especially when dealing with noisy web-crawled data. Their filtering methods significantly boosted BLEU scores.
- NAS Shows Promise: The Transformer-Evolved model, discovered via NAS, achieved competitive performance, suggesting that automated architecture search can rival or even surpass human-designed architectures.
- Back-Translation and KD are Powerful Augmenters: These techniques, when applied strategically, significantly improved translation quality, particularly when combined with careful data filtering.
- R2L Decoding Benefits Ja→Zh: The authors observed that R2L decoding performed better for Japanese-Chinese translation, potentially due to the SOV (Subject-Object-Verb) structure of Japanese, contrasting with the SVO structure of Chinese where L2R is generally preferred.
- Reranking Adds Value: The reranking approach, even on top of strong ensemble models, provided a significant boost in BLEU score for Chinese-Japanese, highlighting its utility in refining translation outputs.
Key Results
- Chinese-Japanese Primary System: Ranked second in character-level BLEU on the official evaluation set.
- Japanese-Chinese Primary System: Achieved competitive performance.
- Data Filtering Impact: Data filtering alone improved BLEU scores by up to 4.70 for Zh→Ja and 5.24 for Ja→Zh.
- Reranking Improvement: The reranking model achieved a significant improvement of about 1.4 BLEU score for Chinese-Japanese.
- Overall Improvement: The final systems achieved substantial improvements of about 9 BLEU score over baseline systems.
Limitations and Future Work
- Translationese Reference Bias: The authors hypothesize that the marginal improvement of reranking for Japanese-Chinese might be due to "translationese" and non-native reference data, which can skew evaluation metrics. This suggests a need for better reference data or evaluation metrics.
- Human-Designed vs. NAS Architectures: While NAS showed promise, the paper doesn't definitively conclude that NAS is always superior. Further research could explore more systematic comparisons and integration of NAS-discovered architectures.
- Interpreting Marginal Improvements: The authors note that it's not entirely clear why certain techniques (like KD for Zh→Ja) yielded only marginal improvements, suggesting areas for deeper analysis.
Future work could involve exploring more advanced data filtering techniques, investigating the impact of different NAS approaches, and developing more robust evaluation methods that are less susceptible to reference bias.
Summary updated