TL;DR
This paper introduces a suite of parallel corpus filtering methods designed to improve the quality of noisy datasets for Neural Machine Translation (NMT). By applying a multi-stage filtering process, Tilde significantly boosts NMT performance, demonstrating that data quality is paramount for achieving high translation accuracy.
What Problem Does This Paper Address?
Training effective Neural Machine Translation (NMT) systems relies heavily on large, high-quality parallel corpora. However, real-world parallel data, often sourced from web crawls like ParaCrawl, is notoriously noisy. This noise can manifest as misalignments, non-translations, or even entirely unrelated sentences, severely hindering NMT model training and leading to poor translation quality. While traditional Statistical Machine Translation (SMT) systems were somewhat robust to this noise, NMT models are far more sensitive. The challenge, therefore, is to develop robust and effective methods to clean these noisy corpora to a level suitable for NMT training, moving from unusable datasets (below 10 BLEU) to high-performing ones (over 20-30 BLEU).
Methods
The authors present a multi-stage filtering pipeline, combining existing methods with novel additions tailored for NMT. The process starts with standard filters designed to remove obvious noise:
- Identical Source/Target Filter: Removes pairs where sentences are identical.
- Length Ratio Filters: Filters pairs where sentence lengths (character or token counts) differ drastically, suggesting partial or incorrect translations.
- Length and Word Count Filters: Removes sentences exceeding maximum character or word limits.
- Unique Sentence Pair Filter: Ensures deduplication, even though the initial dataset was claimed to be deduplicated.
- Foreign Word Filter: Identifies sentences containing words not belonging to the expected language alphabets.
Following these, Tilde introduces additional filters specifically for NMT, aiming for stricter cleaning:
- Empty Sentence Filter: Removes empty sentences.
- Token Count Ratio Filter: A stricter version of the length ratio filter.
- Corrupt Symbol Filter: Detects and removes words with encoding errors (e.g., "flie?en").
- Digit Mismatch Filter: Ensures consistency in how numbers are represented (digits vs. words) between source and target.
- Invalid Character Filter: Removes characters indicative of encoding corruption.
- Invalid Language Filter: Uses language detection to ensure sentences are in the correct languages.
- Stricter Sentence Length Ratio Filter: A more aggressive length ratio filter.
- Low Content Overlap Filter: Utilizes the MPAligner tool to measure content similarity, filtering out pairs with low overlap.
Finally, two additional filters were introduced to address remaining noise, particularly "many-to-many" alignments and untranslated segments:
- Non-translated Sentence Filter: Validates if a significant portion of source words are translated into the target.
- Maximum Alignment Filter: Ensures sentence pairs are reciprocal best matches based on content overlap scores.
The paper also describes a "rescoring" method for the Max Filtered+ dataset, which re-ranks sentence pairs based on length and content overlap to mitigate biases introduced by earlier filters.
Datasets
The primary dataset used for filtering experiments was a raw, deduplicated subset of the German-English ParaCrawl corpus, comprising one billion words and over 104 million sentence pairs. For training and evaluating NMT systems, the authors created several filtered datasets:
- Filtered: Corpus filtered up to the "Low Content Overlap Filter."
- Max Filtered: Corpus filtered using all filters except the "Non-translated Sentence Filter."
- Filtered+: Corpus filtered up to and including the "Non-translated Sentence Filter."
- Max Filtered+: Corpus filtered using all filters.
- Max Filtered+ Rescored: The "Max Filtered+" corpus with sentence pairs rescored.
These filtered datasets were then subsampled into 10 million and 100 million word datasets for NMT training.
Key Insights and Ideas
- NMT's Sensitivity to Noise: The paper strongly reinforces that NMT models are significantly more sensitive to parallel corpus noise than SMT systems, making aggressive filtering crucial.
- Quality Over Quantity: The results demonstrate that a smaller, cleaner corpus (e.g., Filtered+ with ~13.7 million pairs) can yield much better NMT performance than a larger, noisier one.
- Multi-stage Filtering is Key: No single filter is sufficient. A combination of diverse filtering techniques is necessary to effectively clean noisy parallel data.
- "Rescoring" Can Be Detrimental: The "Max Filtered+ Rescored" dataset, despite its intention to correct biases, surprisingly led to a performance drop, especially for smaller datasets. This highlights the difficulty in creating universally optimal scoring mechanisms and the potential for introducing new noise.
- Content Overlap Filter's Limitations: The content overlap filter, while useful, doesn't inherently distinguish reciprocal translations, potentially pairing non-translated segments.
Key Results
- NMT systems trained on the Max Filtered+ dataset achieved the best BLEU scores.
- For the 10 million word experiments, Max Filtered+ yielded a BLEU score of 21.95, a substantial improvement over baseline systems (Hunalign Baseline: 0.15, Random Baseline: 8.41).
- For the 100 million word experiments, Max Filtered+ achieved a BLEU score of 30.83, significantly outperforming baselines (Hunalign Baseline: 3.64, Random Baseline: 7.26).
- The Max Filtered+ Rescored dataset, while showing lower OOV rates, resulted in decreased BLEU scores, indicating that the rescoring process introduced issues, particularly for smaller datasets where longer sentences were less frequent.
- The filtering process significantly reduced Out-of-Vocabulary (OOV) rates, with more aggressive filtering leading to lower OOV percentages.
Limitations and Future Work
- Rescoring Issues: The authors acknowledge that the rescoring method for the "Max Filtered+ Rescored" dataset led to a performance decrease, especially for smaller datasets. They suggest that future work could investigate whether a fixed content overlap threshold might improve the rescoring process.
- Ablation Studies: The authors propose performing ablation experiments to identify which individual filtering methods contribute most to acquiring a higher quality parallel corpus.
- Language Pair Specificity: Some filters, like the sentence length ratio filter, were tested only for specific alphabet types (Latin, Cyrillic, Greek). Further investigation into their applicability across a wider range of languages could be beneficial.
Summary updated