TL;DR
This paper details ISTIC's submission to the IWSLT 2020 Japanese-Chinese and Chinese-Japanese translation tasks. They employed a Transformer-based NMT system, focusing on robust data preprocessing, including Elasticsearch-based filtering for similar corpus selection, and a model averaging strategy for decoding to enhance translation quality.
What Problem Does This Paper Address?
The paper addresses the challenge of building high-quality neural machine translation (NMT) systems for low-resource language pairs, specifically Japanese-Chinese and Chinese-Japanese, within the context of the IWSLT 2020 evaluation. Building effective NMT systems requires careful consideration of training data quality and quantity, model architecture, and decoding strategies. The authors aim to improve translation performance by meticulously refining these aspects, particularly focusing on leveraging noisy web-crawled data and ensuring consistency between development and test sets.
Methods
The core of ISTIC's system is the Transformer architecture (Vaswani et al., 2017), a standard choice for NMT that relies on self-attention mechanisms, eschewing recurrent or convolutional layers for better parallelism and handling of long-distance dependencies.
Key methodological contributions include:
- Data Preprocessing: A two-stage approach was used:
- General Preprocessing: Standard techniques like traditional Chinese to simplified Chinese conversion, word segmentation, filtering of adjacent similar sentences (using Dice similarity), sentence length filtering, and language token ratio filtering were applied.
- Specific Preprocessing: For noisy web-crawled data, they employed GIZA++ to obtain alignment dictionaries and calculated an "alignment ratio" to filter sentence pairs.
- Elasticsearch Similar Corpus Filtering: To improve the consistency between development and test sets, they used Elasticsearch to retrieve similar sentence pairs from the training data (Data 1 and Data 2) for each sentence in the test set. This created new "ES development" and "ES test" sets.
- Model Averaging: To reduce parameter instability and improve robustness, the parameters from the last N (set to 5) epochs of a converged model were averaged.
- Candidate Translations Merge: For the Chinese-Japanese task, where Data 1 introduced noise, they used a primary system trained on Data1+Data2 and a secondary system trained on Data2 alone. A post-processing step merged translations, prioritizing the secondary system if the primary system's output was identical to the source or judged as non-Japanese.
Datasets
The primary datasets used were those released by the IWSLT 2020 organizer for the Open Domain Translation track:
- Data 1:
web_crawled_parallel_filtered.tar.gz - Data 2:
existing_parallel.tar.gz(considered higher quality) - Data 3:
web_crawled_parallel_unfiltered.tar.gz - Data 4:
web_crawled_unaligned.tar.gz(not used due to time constraints)
They also created new development and test sets, referred to as ES development set and ES test set, by using Elasticsearch to find similar sentence pairs from the training data for the original test set sentences.
Key Insights and Ideas
- Leveraging Noisy Data: The paper demonstrates that even noisy web-crawled data (Data 1) can be beneficial when combined with cleaner data (Data 2), provided effective filtering strategies are employed.
- Development Set Importance: The creation of the "ES development set" proved valuable. It showed that using this set, either alone or in conjunction with the original development set, led to better machine translation system performance compared to using the original development set alone. This highlights the importance of development set quality and its alignment with the test set.
- Model Averaging for Robustness: Applying model averaging at decoding time is a practical technique to stabilize NMT models and improve their generalization, especially when dealing with potentially unstable training dynamics.
- Post-processing for Error Correction: The candidate translation merge strategy, particularly for the noisy Chinese-Japanese task, shows that targeted post-processing can effectively mitigate issues arising from data quality problems, such as source-target sentence duplication.
Key Results
- The paper reports BLEU scores for various training data combinations and development set strategies. For instance, in the Chinese-to-Japanese task, using Data1+Data2 resulted in a token BLEU score of 15.83, which improved to 17.64 when Data3 was also included.
- The ES development set consistently outperformed the original development set in Chinese-to-Japanese translation tasks, showing improvements in both token and character BLEU scores. For example, the ES development set achieved a token BLEU of 21.15 compared to 20.79 for the original development set.
- The combined system, after applying data filtering, model averaging, and candidate translation merging, achieved improved translation quality, as evidenced by the qualitative examples and quantitative results presented in the tables.
Limitations and Future Work
- Limited Time: The authors acknowledge that due to time constraints, they could not explore all potential methods, such as using Data 4 or performing extensive consistency analysis on data domains.
- Data 3 Quality: The paper notes that Data 3, sourced from the public network, has limited quality despite its large scale, which might have capped the performance improvements.
- Future Work: The authors express a desire to explore more advanced techniques and construct even better machine translation systems in the future, indicating ongoing research and development in this area.
Summary updated