TL;DR
This paper tackles the challenge of filtering noisy parallel corpora for low-resource languages like Nepali and Sinhala. The authors propose a novel scoring scheme based on fuzzy string matching (specifically Levenshtein distance) to identify high-quality sentence pairs, achieving a strong 2nd place in the WMT 2019 shared task.
What Problem Does This Paper Address?
Building effective machine translation (MT) systems, especially for low-resource languages, is heavily reliant on large amounts of clean, parallel training data. However, data crawled from the web, like the Paracrawl corpus, is often extremely noisy. This noise can manifest as incorrect language identification, sentence pairs that aren't translations of each other, or poor-quality translations. Training MT models on such noisy data significantly degrades their performance. The WMT 2019 parallel corpus filtering task specifically aims to develop methods for scoring sentence pairs within these noisy corpora to extract smaller, high-quality subsets for training MT systems.
Methods
The authors' approach involves a multi-stage filtering process:
- Language Identification: They first use a language identification tool (
langid) to filter out sentence pairs where either the source or target sentence is not in the expected language. - Sentence Length Filtering: To align with MT system training practices (which often have token limits), they filter out sentence pairs where either the source or target sentence exceeds 80 tokens.
- Fuzzy String Matching Scoring: For the remaining sentences, they translate the non-English (Nepali or Sinhala) sentence into English. Then, they compute four fuzzy string matching scores between the original English sentence and the translated English sentence using the
fuzzywuzzyPython library. These scores are based on Levenshtein distance and include: - Ratio (R₁): A standard ratio of matching characters.
- Partial Ratio (R₂): Similar to Ratio, but focuses on the best matching substring.
- Token Sort Ratio (R₃): Compares tokenized strings after sorting the tokens alphabetically.
- Token Set Ratio (R₄): Compares tokenized strings after removing duplicate tokens and sorting.
- Score Combination: These four scores are then combined using both arithmetic mean (Score AM) and geometric mean (Score GM) to produce a final score for each sentence pair.
Datasets
The paper utilizes two main noisy parallel corpora for the WMT 2019 task:
- Nepali-English: Approximately 40.6 million English tokens.
- Sinhala-English: Approximately 59.6 million English tokens.
These corpora are derived from the Paracrawl project. For training the phrase-based SMT systems used in their approach (for translation and evaluation), they also leverage publicly available data sources, including:
- Nepali-English: Bible corpus, Global Voices, Penn Tree Bank, GNOME/KDE/Ubuntu handbooks.
- Sinhala-English: OpenSubtitles, GNOME/KDE/Ubuntu handbooks.
The paper also mentions development (dev) and development test (devtest) sets provided by the task organizers, with specific sentence counts for each language pair.
Key Insights and Ideas
- Fuzzy Matching as a Proxy for Translation Quality: The core idea is that if a non-English sentence, when translated back to English, closely matches the original English sentence using fuzzy string metrics, it's likely a good translation pair. This is a clever way to leverage existing string similarity tools for a translation quality assessment task.
- Low-Resource Languages are Particularly Sensitive to Noise: The paper highlights that MT systems, especially for low-resource languages where data is scarce, are highly sensitive to noise. This underscores the importance of robust filtering techniques.
- Combining Multiple Fuzzy Metrics is Beneficial: Using a combination of different fuzzy string matching ratios (Ratio, Partial Ratio, Token Sort Ratio, Token Set Ratio) likely captures various aspects of string similarity, leading to a more robust scoring mechanism than any single metric alone.
- Initial Filtering is Crucial: The initial steps of language identification and sentence length filtering significantly reduce the corpus size, making subsequent, more computationally intensive fuzzy matching more efficient.
Key Results
The authors' fuzzy string matching scoring scheme achieved strong results in the WMT 2019 parallel corpus filtering task:
- Team Ranking: They secured 2nd position in the team rankings for two categories:
- 1-million Nepali-English NMT
- 5-million Sinhala-English NMT
- SMT Evaluation: When using their sub-sampled sets to train SMT systems, they achieved BLEU scores of 4.20 and 3.50 for Nepali-English (1M and 5M respectively) and 4.20 and 5.10 for Sinhala-English (1M and 5M respectively) on the devtest set, outperforming the baseline in some cases.
Limitations and Future Work
- Language Identifier Imperfections: The paper notes that the language identifier was not perfectly efficient, misclassifying some Nepali sentences as Hindi or Marathi. This suggests room for improvement in the initial language filtering stage.
- Translation Quality Dependency: The fuzzy string matching approach relies on the quality of the intermediate translation from the non-English language to English. If this translation is poor, the fuzzy matching score might be misleading.
- Future Work: While not explicitly detailed, the success in the WMT task suggests that further refinement of the fuzzy matching metrics, their combination, or exploring alternative translation models for the intermediate step could be fruitful avenues. The paper also implicitly points to the need for better language identification tools for these low-resource languages.
Summary updated