Prompsit’s submission to WMT 2018 Parallel Corpus Filtering shared task

Víctor M. Sánchez-Cartagena, Marta Bañón, Sergio Ortiz-Rojas, Gema RamírezEMNLP 2018

TL;DR

This paper details Prompsit Language Engineering's approach to the WMT 2018 parallel corpus filtering task, focusing on building high-quality training data for machine translation. They developed a multi-stage pipeline involving hard rules, an automatic classifier, and novel scoring strategies for vocabulary diversity and fluency, achieving competitive results on large-scale datasets.

What Problem Does This Paper Address?

Building effective machine translation (MT) systems, especially Neural Machine Translation (NMT), relies heavily on the quality of the training data. The WMT 2018 parallel corpus filtering shared task presented a significant challenge: selecting a clean subset of 10 million or 100 million words from a massive, noisy 1-billion-word German-English parallel corpus crawled from the web. The goal was to create training data that would maximize the translation quality of downstream MT systems, evaluated on diverse domains. This requires identifying and discarding sentence pairs that are not true translations, are not fluent, or lack vocabulary diversity, all while dealing with the sheer scale of the data.

Methods

Prompsit's strategy involved a sequential pipeline:

  1. Hard Rule Filtering: Initial cleaning using the Bicleaner tool to remove obvious errors like incorrect language identification, encoding issues, or drastically different sentence lengths.
  2. Automatic Classification: A Random Forest classifier was trained to identify sentence pairs that are mutual translations. This classifier utilized a rich set of features, including:
  3. Lexical Similarity: Based on probabilistic bilingual dictionaries (e.g., DICT-QMAX-L1, DICT-QMAX-L2) to measure how well words in one sentence translate to words in the other.
  4. Shallow Features: Properties like sentence length distribution, number of capitalized words, punctuation marks, and numerical expressions.
  5. Scoring for Fluency and Diversity: Four distinct scoring strategies were explored to refine the corpus, addressing vocabulary diversity and sentence fluency:
  6. N-gram Saturation: This method aimed to increase vocabulary diversity by identifying and discarding sentence pairs that were too similar to others already selected. It used a placeholder-based token replacement strategy to focus on structural similarity rather than specific words.
  7. Active Learning Data Selection: An adaptation of active learning was used to select sentences that would maximize vocabulary diversity. It iteratively selected sentence pairs based on a "Geom n-gram" function that favored sentences with n-grams frequent in the overall corpus but infrequent in the already selected set.
  8. Language Modeling: To ensure fluency, language models (5-gram KenLM) were used to score sentence pairs based on perplexity. Sentences with higher perplexity (indicating less fluency) were penalized. This approach also employed a placeholder strategy to avoid bias from specific vocabulary.

Datasets

  • Training Data: A 1-billion-word German-English parallel corpus (from the Paracrawl project) was the source.
  • Classifier Training: A subset of 60,000 sentences from the news-commentary-v13 corpus was used, with 50,000 for training and 10,000 for validation. Negative instances were synthetically generated by shuffling sentence sides.
  • Language Model Training: 20,000,000 sentences randomly chosen from news and Europarl monolingual corpora were used.
  • Evaluation: The MT systems were evaluated on the newstest2016 (development) and newstest2017 (test) datasets.

Key Insights and Ideas

  • The Importance of Multi-Stage Filtering: Combining hard rules, a classifier, and sophisticated scoring strategies is crucial for handling noisy web-crawled data.
  • Vocabulary Diversity vs. Fluency Trade-off: The paper highlights a tension between maximizing vocabulary diversity and ensuring sentence fluency. For smaller datasets (10M), fluency seemed to have a stronger positive impact on translation quality, while for larger datasets (100M), diversity became more critical.
  • Placeholder Strategy for Robustness: Using placeholders for different token types (e.g., ALPHA:UPPER, NUMERIC) in feature engineering and language modeling proved effective in making the methods robust to variations in proper nouns, numbers, and capitalization, focusing on the underlying sentence structure and translation equivalence.
  • NMT Sensitivity to Noise: The results strongly suggest that NMT systems are significantly more sensitive to noisy training data than SMT systems, as evidenced by the low scores of the "random" baseline for NMT.

Key Results

  • Prompsit's submissions achieved competitive results, ranking close to the top performers for the 100M dataset.
  • The "prompsit-lm-nota" submission (language model scoring without placeholder replacement) achieved the highest BLEU scores for both SMT and NMT on the 100M dataset (24.36 for SMT, 29.85 for NMT).
  • The "prompsit-sat" submission (n-gram saturation for diversity) also performed well, showing the benefit of vocabulary diversity.
  • The active learning approach ("prompsit-al") showed mixed results, with a degradation on the 10M dataset but a slight improvement on the 100M dataset, suggesting its effectiveness might be more pronounced with larger data.
  • The classifier achieved a high accuracy of 0.98 with a score threshold of 0.5.

Limitations and Future Work

  • 10M Dataset Performance: The strategies that prioritized vocabulary diversity or fluency were less successful for the smaller 10M dataset, indicating that the balance between these factors needs careful tuning for different data sizes.
  • Active Learning Tuning: The authors suggest that the active learning algorithm could be further tuned for smaller datasets by decreasing the block size parameter in its initial iterations.
  • Balancing Fluency and Diversity: A future research direction is to develop a scoring scheme that better balances fluency and vocabulary diversity, as this appears to be key for improving results, especially on smaller datasets.

Summary updated

BibTeX
@inproceedings{sanchezcartagena2018prompsits,
  author = {Víctor M. Sánchez-Cartagena and Marta Bañón and Sergio Ortiz-Rojas and Gema Ramírez},
  title = {{Prompsit’s submission to WMT 2018 Parallel Corpus Filtering shared task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure