TL;DR
This paper presents a robust system for filtering large parallel corpora, crucial for training high-quality machine translation models. The RWTH Aachen team combined rule-based heuristics with sophisticated language and translation models to achieve state-of-the-art results in the WMT 2018 parallel corpus filtering task, demonstrating significant improvements in translation quality.
What Problem Does This Paper Address?
Training effective machine translation (MT) systems, especially neural MT (NMT) models, relies heavily on the quality and quantity of parallel data (sentences aligned in two languages). However, publicly available parallel corpora, like the massive ParaCrawl dataset, often contain a significant amount of "noise" – misaligned sentences, non-parallel content, or repetitive text. This noise can severely degrade MT performance. The challenge is to efficiently and effectively filter these large datasets to retain only high-quality parallel sentence pairs, a task that is computationally expensive when using complex NMT models directly.
Methods
The RWTH Aachen University system employs a two-stage filtering approach:
- Rule-Based Heuristics: This initial stage rapidly prunes the massive corpus (104 million sentence pairs) using a series of language-agnostic rules. These include:
- Minimum Words: Ensuring sentences have a minimum number of actual words (not just numbers or punctuation).
- Average Word Length: Removing sentences with unusually long or short "words" (e.g., long URLs).
- Length Ratio: Filtering pairs where the source and target sentence lengths differ significantly.
- Maximum Sentence Length: Limiting sentence length to avoid issues with model capacity and computational cost.
- Maximum Subword-Token Length: Enforcing a limit on BPE (Byte Pair Encoding) subword units, necessary for transformer models.
- Levenshtein Distance: Identifying and removing sentence pairs where the source and target are too similar at the word level, preventing models from simply "copying" text.
- Word Token Ratio: A heuristic to remove sentences with a high proportion of numbers and punctuation.
- Redundancy Removal: A novel approach to remove near-duplicate sentences by hashing variations of sentences with single words removed.
- Model-Based Scoring: After the rule-based filtering, the remaining sentences are scored using various models to establish a ranking. The paper explores:
- Count-Based Language Models (KenLM): Standard n-gram models trained on monolingual data.
- IBM1 Dictionary Model: A simple alignment model.
- Neural Network Language Models (LSTM-based): Recurrent neural networks trained as language models.
- Transformer Translation Models: State-of-the-art NMT models used for scoring.
The final submission combines these models, often using a weighted average of their scores, to rank the filtered sentence pairs.
Datasets
- Training Data: The primary dataset used for filtering and model training is the WMT 2018 German-English parallel corpus, which includes sources like Europarl, CommonCrawl, NewsCommentary, and Rapid. Additional monolingual data from NewsCrawl 2012-2015 was also used.
- Evaluation Data: The system's performance was evaluated on the WMT 2018 test sets (newstest2015, newstest2017, newstest2018). The paper also mentions using newstest2015 for checkpoint selection.
Key Insights and Ideas
- Heuristics are Powerful and Fast: The rule-based heuristics, while simple, are surprisingly effective at removing a large portion of noisy data and significantly improving translation quality, even before applying complex models.
- Levenshtein Distance is Crucial for NMT: The Levenshtein distance heuristic proved particularly important for neural models, which tend to over-assign high scores to sentences that are too similar between source and target, essentially rewarding copying.
- Redundancy Removal Adds Value: The novel redundancy removal heuristic further boosted performance, suggesting that even subtle similarities between sentences can be detrimental.
- Language Models vs. Translation Models for Scoring: While language models alone provided some cleaning, translation models (especially neural ones) were more effective at identifying and scoring parallel sentence quality.
- Noisy Training Data Hurts: Adding noisy data (like CommonCrawl) to the training set for scoring models, even if it improves perplexity, can actually decrease translation quality because the models learn to assign higher scores to bad sentences.
Key Results
The RWTH Aachen University system achieved strong results in the WMT 2018 parallel corpus filtering task.
- The rule-based filtering alone (up to step 3.6) reduced the corpus by 60% and improved BLEU scores on test sets by over 10% compared to random sampling.
- The final submission, combining rule-based filtering with a sophisticated model-based scoring system (KenLM + IBM + BlocksLM + Transformer), achieved a performance of 34.8% BLEU on newstest2018. This is a significant improvement over the baseline of 9.2% BLEU achieved with random sampling on a 10M token subsampled dataset.
- The paper reports being slightly ahead of Microsoft's submission on the SMT 10M evaluation and slightly behind on NMT 100M, but achieving the best results on newstest2018 for NMT 10M.
Limitations and Future Work
- Domain Adaptation: The authors note that performance might be weaker on domains different from the training data due to potential issues with domain adaptation when training language models.
- Subword Tokenization Issues: The language-specific tokenizers used in preprocessing could sometimes lead to inconsistencies if the same patterns were not split identically across source and target, potentially affecting distance metrics.
- Evaluation Metric Sensitivity: The paper acknowledges that their BLEU scores might vary compared to other submissions due to specific tokenization and subword procedures, which can increase the number of tokens per sentence.
Future work could explore more advanced domain adaptation techniques and investigate the impact of different subword segmentation strategies on filtering performance.
Summary updated