Webinterpret Submission to the WMT2019 Shared Task on Parallel Corpus Filtering

Jesús González-RubioACL 2019

TL;DR

This paper details WebInterpret's submission to the WMT2019 shared task on parallel corpus filtering, aiming to clean noisy web-crawled text for machine translation. They propose a novel approach that prioritizes sentence pairs offering the most coherent coverage without relying on pre-trained models, a key advantage for low-resource languages.

What Problem Does This Paper Address?

Machine translation systems thrive on high-quality parallel corpora – pairs of sentences that are accurate translations of each other. However, web-crawled data, while abundant, is often noisy, containing misaligned sentences, incorrect translations, or sentences that are too dissimilar. The WMT2019 shared task on parallel corpus filtering challenged participants to develop methods for cleaning these noisy corpora, specifically focusing on low-resource languages like Nepali and Sinhala, and producing smaller, high-quality subsets for training MT systems. The goal is to improve translation quality by removing bad sentence pairs, rather than just increasing the sheer volume of data.

Methods

WebInterpret's approach combines rule-based filtering with more sophisticated ranking mechanisms.

  • Rule-based Filtering: Initially, they apply straightforward filters:
  • Language Identification: Ensures both sentences in a pair are in the target language pair (e.g., Nepali-English).
  • Length Ratio: Filters pairs where the sentence lengths (token counts) differ significantly, as this often indicates misalignment.
  • Maximum Sentence Length: Removes pairs where either sentence exceeds a predefined length, as many MT systems ignore very long sentences.
  • Coverage Ranking: This is their core novel contribution. Instead of relying on in-domain data to identify "underrepresented" words or phrases, they aim to find sentence pairs that, when added to a growing training corpus, contribute to the most coherent overall coverage. They use a greedy algorithm to iteratively select sentences that maximize a "value" function, which measures how well different tokens (n-grams up to size 4) are represented in the corpus.
  • Adequacy Ranking: To complement coverage, they also implement an "adequacy" ranking. This measures how well the meaning of one sentence is preserved in its translation. They use a simple word-to-word translation model (IBM Model 1) and its Viterbi variant to estimate lexical probabilities between words in source and target sentences.
  • Ranking Aggregation: Finally, they combine the coverage and adequacy rankings. They average multiple coverage rankings (for different n-gram sizes and source/destination directions) and multiple adequacy rankings (from different model variants) to get a single coverage score ($r_c$) and a single adequacy score ($r_a$). The final ranking ($r$) for a pair is the average of these two. The submission score is then derived from this final ranking.

Datasets

The primary datasets used were the noisy Nepali-English and Sinhala-English corpora provided by the WMT2019 organizers, which were crawled as part of the Paracrawl project. These corpora were substantial (40.6 million English tokens for Nepali-English and 59.6 million for Sinhala-English). The evaluation was performed using held-out Wikipedia translation test sets.

Key Insights and Ideas

  • Coverage without Domain Data: The most significant idea is developing a coverage ranking that doesn't require pre-existing in-domain data. This is crucial for low-resource languages where such data is scarce.
  • Greedy Algorithm's Potential Pitfalls: The greedy approach to coverage ranking, while intuitive, might make suboptimal initial choices that the algorithm cannot recover from, potentially explaining some of the observed performance issues.
  • NMT Degradation with More Data: A surprising observation across multiple submissions (including their own) was that NMT performance sometimes decreased when more data was added, contrary to the usual expectation that more data improves translation quality. This suggests that simply adding more noisy data can be detrimental.

Key Results

WebInterpret submitted three versions of their system: PRIMARY (combined coverage and adequacy), SECONDARYCOV (coverage only), and SECONDARYADE (adequacy only).

  • SECONDARYADE Outperformed: Across both language pairs and both 1 million and 5 million word conditions, the SECONDARYADE submission (adequacy only) generally performed best, followed by PRIMARY. The SECONDARYCOV (coverage only) submission consistently underperformed.
  • Coverage Ranking's Weakness: The paper notes that their proposed coverage approach was "not well suited for this particular task," especially compared to previous work that assumed in-domain data.
  • Competitive Performance: Their PRIMARY submission scored 7.1 and 6.7 BLEU points for Nepali-English (1M and 5M conditions, respectively) and 6.8 and 5.8 for Sinhala-English. These scores represented a significant portion (around 64-85%) of the best results submitted by other participants.

Limitations and Future Work

  • Coverage Approach Limitations: The authors acknowledge that their coverage ranking approach, particularly the greedy selection, might not be optimal and could be sensitive to initial decisions. They hypothesize that the lack of in-domain data and the eclectic nature of the web-crawled data contributed to its poor performance.
  • NMT Data Sensitivity: The observed degradation in NMT performance with more data is highlighted as an area for future research. Developing methods to identify when adding more data becomes harmful is suggested as a promising direction.
  • Time Performance: The initial greedy coverage algorithm had a high computational cost ($O(R^2)$). They mitigated this by implementing a batch approach (selecting 1000 sentences at a time), which significantly reduced runtime with minimal impact on corpus "value." Further optimization of this algorithm is implied.

Summary updated

BibTeX
@inproceedings{gonzalezrubio2019webinterpret,
  author = {Jesús González-Rubio},
  title = {{Webinterpret Submission to the WMT2019 Shared Task on Parallel Corpus Filtering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure