TL;DR
This paper presents ILSP/ARC's approach to the WMT 2018 Parallel Corpus Filtering task, focusing on a language-agnostic method to cluster sentence pairs by quality for training machine translation systems. Their system achieved a respectable BLEU score without relying on external language-specific resources, highlighting the effectiveness of their unsupervised filtering techniques.
What Problem Does This Paper Address?
Building high-quality parallel corpora, essential for training machine translation (MT) systems, is a significant challenge. While monolingual data is abundant on the web, identifying and extracting accurate sentence pairs that are actual translations of each other is difficult. This process is further complicated by "noise" introduced during web crawling and parsing, such as misaligned sentences, boilerplate text, and non-target language content. The WMT 2018 Parallel Corpus Filtering task specifically aimed to clean a massive, noisy English-German corpus, requiring participants to develop methods for ranking sentence pairs by their suitability for MT training.
Methods
The ILSP/ARC submission employed a two-stage, language-agnostic filtering approach.
- Filter-based Clustering: The first stage involved clustering sentence pairs into five categories based on shallow, language-independent features. This included identifying obviously noisy pairs (e.g., too short/long sentences, non-letter content, URLs, identical text) and assigning them a score of 0. The remaining pairs were then clustered based on the similarity of their digit and symbol patterns (punctuation, etc.), creating four clusters ordered by presumed quality.
- Near-Duplicate Identification: In the second stage, the system identified and handled near-duplicate sentence pairs. When duplicates were found, the pair belonging to a higher-quality cluster was retained. If both pairs were in the same cluster, the longer one was kept.
The system then used these cluster assignments to rank sentence pairs. Two ranking schemes were explored: one directly using cluster numbers as scores, and another combining cluster scores with the Hunalign sentence aligner's score. A third, alternative ranking scheme incorporated sentence length information.
Datasets
The primary dataset used was the noisy English-German parallel corpus of 104 million sentence pairs provided by the WMT 2018 Parallel Corpus Filtering shared task organizers. This corpus was accompanied by Hunalign scores for each sentence pair.
The evaluation used standard WMT test sets, including the official WMT 2018 news translation test set and an undisclosed test set comprising a union of five other test sets.
Key Insights and Ideas
- Language-Agnostic Filtering is Viable: The paper demonstrates that effective parallel corpus filtering can be achieved without relying on language-specific resources like bilingual lexicons or pre-trained MT models, which are often scarce.
- Shallow Features Capture Quality: Simple features related to sentence length, character types, and pattern matching proved surprisingly effective in identifying and clustering sentence pairs by their potential usefulness for MT training.
- Noise is Multifaceted: The study highlights the diverse nature of noise in web-crawled data, ranging from structural issues (HTML parsing) to content-related problems (non-target languages, boilerplate).
- Clustering as a Proxy for Quality: The clustering approach effectively groups sentence pairs, providing a ranked list that can be used to select subsets for MT training.
Key Results
The ILSP/ARC submission achieved an average BLEU score of 26.41 across various test configurations. This was competitive, though slightly lower than the best participant system's average of 27.91. Notably, their system did not utilize any language-specific resources.
When comparing the two main ranking schemes:
- The cluster-based ranking (combined with Hunalign) yielded an average BLEU score of 26.41.
- The alternative ranking scheme (using sentence length) resulted in an average BLEU score of 24.98.
The evaluation also showed that Neural Machine Translation (NMT) systems generally outperformed Statistical Machine Translation (SMT) systems when trained on the filtered corpora, with larger training corpora (100M tokens) leading to significant BLEU score improvements for both SMT and NMT.
Limitations and Future Work
- Limited Granularity of Scores: The initial clustering approach assigned only a few distinct scores, which could lead to the selection of more data than required for the 10M and 100M word thresholds. This was addressed by refining the ranking scheme.
- Impact of Corpus Size vs. Sentence Length: The paper notes that for NMT systems, the size and length of sentence pairs in the training corpus can have a significant impact, and suggests future work to investigate this relationship further.
- Reconsidering Specific Patterns: The authors mention that the usefulness of certain filtering patterns (e.g., for preparing parallel corpora for NMT engines) warrants further reconsideration in future work.
Summary updated