The Speechmatics Parallel Corpus Filtering System for WMT18

Tom Ash, Remi Francis, Will WilliamsEMNLP 2018

TL;DR

Speechmatics developed a two-pass system to filter massive parallel corpora for machine translation (MT) training. Their approach effectively removes "junk" data and distills the remaining data into smaller, high-quality subsets, significantly improving MT model performance with minimal tuning.

What Problem Does This Paper Address?

Training effective machine translation systems requires vast amounts of high-quality parallel text (sentences and their translations). However, large web-crawled corpora, while abundant, are often noisy, containing misalignments, untranslated sentences, or other forms of "junk" data. Filtering these massive datasets to create smaller, cleaner subsets is crucial for efficient training and improved translation quality. The WMT18 task specifically asked for a system to score sentence pairs from a 1-billion-word corpus, allowing for filtering into 10 million and 100 million word subsets, with performance measured by BLEU scores of MT systems trained on these subsets.

Methods

The Speechmatics system employs a two-pass strategy:

  • Pass 1: Hard Rules: This initial pass uses a series of pragmatic "hard rules" to aggressively remove the worst offenders. These rules include:
  • Line Length: Eliminating sentence pairs with drastically different lengths, based on the assumption that translations should have a similar word count.
  • Non-translation: Removing pairs with a high BLEU similarity score, indicating they are likely not translations of each other.
  • Language Identification: Using a tool (lang-id) to verify that both sentences in a pair are indeed in the expected languages.
  • Character Filtering: Removing sentences containing characters deemed "unacceptable" for a given language, which helps catch incorrect language identification and malformed markup.
  • Digit Matching: Ensuring that numerical digits present in both source and target sentences match, preventing obvious mismatches. This first pass significantly reduces the corpus size from 1 billion to 160 million tokens.

  • Pass 2: Scoring Heuristics: The remaining 160 million tokens are then scored using four heuristics, which are then combined to produce a final score for each sentence pair. These heuristics are designed to be general and not overly tuned to specific language pairs:

  • Sentence Length: Encourages longer sentences, as very short sentences were found to be indicative of lower quality.
  • Perplexity: Measures how "surprising" a sentence is to a language model trained on the data itself. Sentences with perplexity closer to that of a clean "dev" corpus are favored.
  • Diversity: Uses a rolling buffer to measure sentence similarity to its neighbors, favoring sentences that are distinct and add new information.
  • MT Filtering: Trains a machine translation system on the post-rules corpus and uses the decoder cost (cross-entropy loss) of the reference translation and the best translation to identify sentences that are unlikely to be good translations.

Datasets

  • 1-billion-word parallel corpus: The primary dataset for filtering.
  • WMT 2018 English-German development set: A smaller, cleaner corpus (approx. 130M words) used to inform the heuristics and for benchmarking. This set includes data from Europarl, Common Crawl, News Commentary, and Rapid EU Press Release Corpora, excluding the Paracrawl corpus.

Key Insights and Ideas

  • Aggressive initial filtering is highly effective: The "hard rules" in the first pass provide the majority of the BLEU score improvement, demonstrating that removing obviously bad data is paramount.
  • General heuristics are sufficient: The system achieves competitive results without extensive fine-tuning to specific language pairs or tasks, highlighting the robustness of the chosen heuristics.
  • Diversity and perplexity are key for smaller subsets: For the 10 million word subset, diversity and perplexity heuristics were crucial for improving BLEU scores, while sentence length and MT filtering alone degraded performance. This suggests that for very small, high-quality subsets, capturing linguistic variety and naturalness is critical.
  • MT filtering has high potential: The authors believe the MT filtering heuristic has the most scope for future development, especially as model capacity scales, suggesting dynamic filtering during training might be optimal.
  • Character filtering reduces effective vocabulary: Eliminating characters not in a defined "acceptable" list not only cleans data but also implicitly reduces the effective vocabulary size of MT systems, which was found to be beneficial for modern NMT.

Key Results

  • The initial hard rules reduced the 1-billion-word corpus to 160 million words, achieving a significant BLEU score improvement from 5.93 to 26.14 for a 10 million word target subset when compared to random selection from the initial corpus.
  • When filtering down to 100 million words, using all four heuristics together provided a total BLEU gain of 1.97 over random selection from the 160M post-rules corpus.
  • For the 10 million word subset, combining all four heuristics equally yielded a substantial gain of 5.22 BLEU points.
  • The system's best submission achieved a BLEU score of 31.36 on the 10 million word subset and 33.11 on the 100 million word subset.

Limitations and Future Work

  • Parameter Tuning: The authors acknowledge that they did not extensively tune the parameters of their system. Further optimization of the nine components could lead to even better results.
  • Domain Specificity: For realistic use cases, domain-specific entropy filtering would be beneficial, as previously found in language modeling.
  • MT Filtering Development: The MT filtering heuristic is identified as having the most potential for future work, particularly investigating its efficacy with scaled model capacity and exploring dynamic filtering during training.
  • Language Identification False Positives: The language identification rule was noted as having a potential for false positives, especially with short sentences, though it still improved overall corpus quality.

Summary updated

BibTeX
@inproceedings{ash2018speechmatics,
  author = {Tom Ash and Remi Francis and Will Williams},
  title = {{The Speechmatics Parallel Corpus Filtering System for WMT18}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure