Dual Monolingual Cross-Entropy Delta Filtering of Noisy Parallel Data

Amittai Axelrod, Anish Kumar, Steve SlotoACL 2019

TL;DR

This paper introduces a novel, purely monolingual approach to filtering noisy parallel data for low-resource machine translation. Their method, a modification of existing cross-entropy filtering techniques, achieves competitive results with bilingual methods while being significantly more efficient and requiring less parallel data.

What Problem Does This Paper Address?

Training effective machine translation (MT) systems, especially for low-resource languages, heavily relies on large amounts of high-quality parallel data (sentences and their translations). However, publicly available parallel corpora are often noisy, containing misaligned or non-parallel sentence pairs. Traditional methods for filtering this noisy data often rely on bilingual information (e.g., using two MT systems trained on the same data to compare translation probabilities). This paper tackles the challenge of filtering noisy parallel data in low-resource scenarios where obtaining sufficient parallel data to train robust bilingual filtering models is difficult or impossible. They propose a method that leverages monolingual data and a "cynical" data selection approach to achieve effective filtering without requiring extensive parallel data for the filtering process itself.

Methods

The core of the paper's contribution lies in its novel filtering features, which are entirely monolingual. They build upon the work of Junczys-Dowmunt (2018) but adapt it for low-resource scenarios:

  • Dual Monolingual Cross-Entropy Delta Filtering: This is the main innovation. Instead of using bilingual MT models to compare translation probabilities, they use monolingual language models (LMs). They compute the "cross-entropy delta" (ΔH) for each sentence in the noisy corpus with respect to a representative monolingual corpus (e.g., Wikipedia). This ΔH measures how much adding a sentence to the training data of an LM would reduce its perplexity on the representative data. By comparing the ΔH scores from Sinhala LMs and English LMs for a sentence pair, they infer the likelihood of the pair being parallel. This is a relaxation of the bilingual conditional cross-entropy, using "relative informativeness" instead of "equal translation probability."
  • Cynical Data Selection: This approach, adapted from Axelrod (2017), is used to rank sentences. Instead of assuming a stark in-domain/out-of-domain distinction, it starts with a representative corpus (REPR) and an available corpus (AVAIL). It incrementally selects sentences from AVAIL that best improve a language model trained on REPR. For filtering, they use a modified version where the entire noisy corpus is scored in a single pass, ranking sentences based on how much they contribute new information to a representative monolingual corpus.
  • Length Ratio Feature: They introduce a feature that penalizes sentence pairs where the lengths of the two halves are significantly different. This is motivated by the observation that non-parallel sentence pairs often have disparate lengths, and also by the tendency of MT systems to struggle with length mismatches.
  • Language ID Feature: A standard feature that identifies and removes sentence pairs where one or both halves are not in the expected language. They enhance this by multiplying the language ID confidence score by the proportion of characters belonging to the correct Unicode block, making it more robust to mixed-language sentences.

Datasets

  • Parallel Data:
    • WMT 2018 Noisy Parallel Corpus: A large (1B words) German-English corpus used in previous WMT shared tasks.
  • WMT 2019 Sinhala-English Parallel Data: A smaller, noisy corpus (645k lines) drawn from conversations and technical manuals, used for the low-resource track.
  • Monolingual Data:
    • Wikipedia and Common Crawl: Provided large monolingual corpora in Sinhala and English. For filtering, they created balanced monolingual English corpora comparable in size to the Sinhala data by subsampling Wikipedia and Common Crawl.
  • Clean Monolingual Wikipedia Data: Used as the "Representative" corpus for the Cynical data selection approach.

Key Insights and Ideas

  • Monolingual Power: The most significant insight is that effective parallel data filtering can be achieved using only monolingual data and language models, even for low-resource languages. This bypasses the need for expensive and difficult-to-obtain parallel data for the filtering process itself.
  • Informativeness over Probability: The dual monolingual cross-entropy delta approach prioritizes sentences that are informative for a representative language model, rather than just those that are likely translations. This is a crucial shift in perspective for filtering.
  • Cynical Selection's Adaptability: The paper demonstrates that Cynical data selection, originally designed for domain adaptation, can be effectively adapted for filtering noisy parallel data by treating the noisy corpus as the "available" set and a clean monolingual corpus as the "representative" set.
  • Efficiency Matters: The proposed method is "featherweight," running end-to-end on a standard laptop in a few hours, making it highly practical for researchers and practitioners with limited computational resources.

Key Results

  • The proposed monolingual filtering method achieved results competitive with the best bilingual filtering methods in the WMT 2019 low-resource task.
  • Their system was within 1.8 BLEU of the best system at 1M English words and 1.3 BLEU of the best system at 5M English words.
  • Despite being a "gratuitously-monolingual approach to a bilingual task," their system was competitive, highlighting the effectiveness of their monolingual features.
  • The method is computationally efficient, running end-to-end on a standard laptop in three hours.

Limitations and Future Work

  • Length Ratio Feature Complexity: The authors acknowledge that their length ratio feature was "overly complicated and not aggressive enough," leading to MT systems that "stuttered." They suggest that a simpler threshold (e.g., discarding pairs with a length difference > 20%) might have been more effective.
  • Low-Resource NMT Pipeline Suitability: While competitive, their system was not suitable for filtering data for a low-resource NMT pipeline in the context of the WMT 2019 task, as indicated by the performance of the trained NMT systems.
  • Feature Engineering: The authors suggest that "better feature engineering would certainly help" to improve performance, particularly for low-resource NMT pipelines.
  • Integration into Feature Ensembles: The method is presented as well-suited for integration into a feature ensemble for real-world deployment, implying future work could involve combining it with other filtering techniques.

Summary updated

BibTeX
@inproceedings{axelrod2019dual,
  author = {Amittai Axelrod and Anish Kumar and Steve Sloto},
  title = {{Dual Monolingual Cross-Entropy Delta Filtering of Noisy Parallel Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure