MAJE Submission to the WMT2018 Shared Task on Parallel Corpus Filtering

Marina Fomicheva, Jesús González-RubioEMNLP 2018

TL;DR

This paper presents Webinterpret's submission to the WMT2018 shared task on parallel corpus filtering. They frame the problem as a quality estimation task, developing a gradient boosting model to identify high-quality sentence pairs from a massive, noisy web-crawled corpus, crucial for training better machine translation systems.

What Problem Does This Paper Address?

Machine translation (MT) systems thrive on large, clean parallel corpora – collections of sentences and their translations. However, automatically collecting such data from the web often results in massive datasets that are extremely noisy. This noise can include incorrect language pairs, non-translations, or grammatically poor sentences, all of which degrade MT model performance. The WMT2018 shared task specifically aimed to address this challenge by providing a noisy 1-billion-word German-English corpus and asking participants to filter it down to smaller, high-quality subsets (10 million and 100 million words). The quality of the filtered data was then evaluated by training MT systems on it and measuring their performance on blind test sets.

Methods

Webinterpret approached this problem by framing parallel corpus filtering as a Quality Estimation (QE) task. QE, in the absence of reference translations, aims to predict the quality of a translation. Here, they adapted this to estimate how well two sentences in a pair correspond to each other, essentially predicting if they are a valid translation.

Their core method involves:

  • Feature Engineering: They developed a rich set of features to capture sentence pair correspondence, categorized into:
  • Adequacy: Measuring how much meaning is preserved between sentences using probabilistic lexicons (e.g., Average Max lexical probability, Cross-entropy).
  • Fluency: Assessing grammatical correctness and naturalness using language models (e.g., language model scores, perplexity).
  • Shape: Quantifying mismatches in token frequencies and types (e.g., token counts, Jaccard index).
  • Negative Example Generation: Since "bad" sentence pairs aren't readily available, they generated negative examples by perturbing existing sentence pairs. Common methods included swapping source and target sentences, duplicating strings, or replacing a sentence with a random one from the corpus.
  • Classification Model: They experimented with various classifiers and found Gradient Boosting to be the most effective. This ensemble method builds a strong predictive model from many weak decision trees.

Datasets

  • Training Data: The News Commentary V13 parallel corpus was used to train the probabilistic lexicons, language models, and the gradient boosting classifier. This corpus was also used to generate negative examples.
  • Evaluation Data: The primary dataset was the 1-billion-word German-English Paracrawl corpus, provided by the WMT2018 organizers. Participants submitted quality scores for this corpus, which were then used to select subsets for MT system training and evaluation.

Key Insights and Ideas

  • Framing filtering as QE is effective: Treating the problem as a quality estimation task, rather than just a simple filtering mechanism, allowed them to leverage established QE techniques and feature engineering.
  • Synthetic negative examples are crucial: The ability to generate realistic "bad" sentence pairs through perturbation was key to training a robust classifier.
  • Gradient Boosting excels at this task: The choice of gradient boosting proved to be a strong performer, capable of handling the complex feature interactions.
  • Iterative training could improve robustness: While not fully explored due to time constraints, the idea of iteratively training the model and cleaning data holds promise for further improvements.

Key Results

Webinterpret's submission achieved competitive results in the WMT2018 shared task. For the 10 million token sub-sampling, their system scored 22.5 for statistical MT (SMT) and 24.8 for neural MT (NMT). For the 100 million token sub-sampling, they scored 26.1 for SMT and 31.2 for NMT. While these scores were slightly lower than the best submission (24.5/28.6 for 10M and 26.5/32.1 for 100M), they were within a close range, indicating the effectiveness of their approach. Notably, the results for the 100 million token sub-sampling showed less variation between participants, suggesting that larger data sizes tend to neutralize differences in data cleaning methods.

Limitations and Future Work

The authors acknowledge that their approach considers each sentence pair in isolation, which might be sub-optimal. They also note that the evaluation of MT systems on blind test sets makes it difficult to interpret small differences between participant submissions without confidence intervals.

Future work directions suggested include:

  • Neural Models for Feature Extraction: Instead of hand-crafting features, using neural networks to automatically learn relevant features.
  • More Sophisticated Perturbation: Developing new perturbation operations that specifically degrade the "quality" of sentence pairs for training data.
  • Iterative Training Procedures: Implementing a loop where model training and data cleaning are repeated to achieve better convergence and reduce dependency on initial data quality.

Summary updated

BibTeX
@inproceedings{fomicheva2018maje,
  author = {Marina Fomicheva and Jesús González-Rubio},
  title = {{MAJE Submission to the WMT2018 Shared Task on Parallel Corpus Filtering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure