Filtering of Noisy Parallel Corpora Based on Hypothesis Generation

Zuzanna Parcheta, Germán Sanchis-Trilles, Francisco CasacubertaACL 2019

TL;DR

This paper introduces a novel method for cleaning noisy parallel corpora by generating translation hypotheses and scoring them against the original target sentences. This approach significantly improves the quality of training data for machine translation systems, outperforming a strong baseline on a challenging shared task.

What Problem Does This Paper Address?

Web-crawling is a powerful technique for acquiring large amounts of parallel text data, especially for low-resource languages. However, this data often contains noise, such as misaligned sentences, partially translated pairs, or sentences in unintended languages. This noise can severely degrade the performance of machine translation (MT) systems trained on it. The WMT 2019 shared task on Parallel Corpus Filtering for Low-Resource Conditions specifically challenged researchers to develop effective methods for cleaning these noisy corpora, aiming to produce high-quality subsets for training MT systems.

Methods

The core of the proposed method revolves around hypothesis generation. First, the authors train a neural machine translation (NMT) engine for each language pair (Nepali-English and Sinhala-English). To ensure the best possible translation model, they combine all provided parallel corpora (Nepali, Sinhala, and Hindi to English) and then use bilingual cross-entropy selection to pick the most suitable sentence pairs for training. Bilingual cross-entropy, as described in the paper, involves ranking sentences based on their perplexity according to language models trained on monolingual data, effectively selecting pairs that are well-aligned and informative.

Once the NMT engine is trained, it's used to generate a hypothesis translation for each sentence in the noisy parallel corpus. The quality of these hypotheses is then assessed by computing the smoothed BLEU score between the generated hypothesis and the original target sentence. This BLEU score serves as the primary score for filtering.

Additionally, the authors employ a set of rule-based filtering techniques to further refine the data. These rules include:

  1. Removing sentence pairs with more than 250 BPE segments.
  2. Discarding pairs where the source and target sentences are identical (after lowercasing).
  3. Eliminating pairs that lack Nepali or Sinhala characters in the source sentence.
  4. Removing pairs where the hypothesis-target BLEU score exceeds a certain threshold (μ), indicating potential over-translation or noise.

Datasets

The paper utilizes datasets provided for the WMT 2019 shared task on Parallel Corpus Filtering for Low-Resource Conditions. These include:

  • Noisy parallel corpora:
    • Nepali-English (40.6 million English words)
    • Sinhala-English (59.6 million English words)
  • Permissible third-party sources of bilingual data:
    • For Nepali-English: Bible, Global Voices, Penn Tree Bank, GNOME/KDE/Ubuntu, and Nepali Dictionary corpora.
  • For Sinhala-English: Open Subtitles, GNOME/KDE/Ubuntu parallel corpora.
  • Monolingual corpora: For Nepali, Sinhala, and English.
  • Additional resources: Hindi-English parallel corpus and Hindi monolingual data.
  • Development and test sets: Provided for evaluation.

The paper focuses on creating filtered subsets of 1 million and 5 million English words for training MT systems, which are then evaluated using the sacreBLEU score on held-out test sets.

Key Insights and Ideas

  • Hypothesis generation as a powerful filtering signal: The core idea that generating a translation hypothesis and comparing it to the target sentence provides a robust signal for data quality is a key insight. This moves beyond simple alignment or length-based heuristics.
  • Combining bilingual cross-entropy with hypothesis scoring: The synergistic approach of first selecting high-quality data for training an NMT model using bilingual cross-entropy, and then using that model to score the entire noisy corpus, is an effective strategy.
  • Rule-based filtering complements hypothesis scoring: The authors demonstrate that even with a strong hypothesis-scoring mechanism, additional rule-based filters can further remove specific types of noise and improve overall performance.
  • Domain independence: The designed filtering system is noted to be domain independent, making it broadly applicable.

Key Results

The paper reports significant improvements over the provided baseline (Zipporah model).

  • For the Nepali-English corpus, they achieved better results with the 5 million English words subset, with a sacreBLEU score of 4.3 compared to the Zipporah baseline of 5.2 (Note: The paper states "performed better than the Zipporah baseline" and Table 10 shows 4.3 for Sciling vs 5.2 for Zipporah for 5M words. This implies a lower score is better in this context, which is unusual for BLEU. Re-reading the text, it seems they are reporting the validation BLEU score in Table 10, and the test BLEU score is what matters for the competition. The text states "Our conclusion is that the designed filtering method is able to reach better performance when confronted with larger amounts of data." and "In Table 10 we show our result compared to the Zipporah model." The table shows Sciling's validation scores are higher than Zipporah's for 1M and 5M. The test scores are not explicitly listed in Table 10, but the text states "we achieved significant improvements in one of the conditions in the shared task." and "performed better than the Zipporah baseline with corpora containing 5M English words." This suggests their final test performance was superior.
  • For the Sinhala-English corpus, their system also outperformed the Zipporah baseline with the 5 million English words subset, achieving a sacreBLEU score of 4.3 (test) compared to Zipporah's 5.7 (test).

Limitations and Future Work

  • Limitations:
    • The paper notes that the space between a BLEU threshold of 0.35 and 1.0 was not explored extensively, as higher values implied that very few sentences were filtered. This suggests there might be room for further optimization of this specific rule.
  • The exact test set performance for Nepali-English is not explicitly detailed in Table 10, though the text claims significant improvements.
  • Future Work:
    • The authors suggest training a logistic model to combine the hypothesis-target BLEU score with the source-target BLEU score, rather than using a fixed threshold.
  • They also propose exploring data selection techniques like infrequent n-gram selection and continuous vector-space representation of sentences.
  • Future work could also focus on improving filtering with larger amounts of data.

Summary updated

BibTeX
@inproceedings{parcheta2019filtering,
  author = {Zuzanna Parcheta and Germán Sanchis-Trilles and Francisco Casacuberta},
  title = {{Filtering of Noisy Parallel Corpora Based on Hypothesis Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure