STACC, OOV Density and N-gram Saturation: Vicomtech’s Participation in the WMT 2018 Shared Task on Parallel Corpus Filtering

Andoni Azpeitia, Thierry Etchegoyhen, Eva Martínez GarciaEMNLP 2018

TL;DR

This paper presents Vicomtech's participation in the WMT 2018 Shared Task on parallel corpus filtering. They propose a simple, efficient, and portable method based on the STACC system, augmented with an Out-Of-Vocabulary (OOV) density penalty and an n-gram saturation technique, achieving competitive results without requiring prior training.

What Problem Does This Paper Address?

Modern machine translation (MT) systems, especially Neural Machine Translation (NMT), thrive on vast amounts of parallel text data. However, acquiring large, high-quality parallel corpora is challenging. While the web offers a rich source of multilingual data, automatically collected corpora often suffer from significant noise, including misaligned sentences, incorrect language identification, and untranslated segments. This noisy data can severely degrade MT system performance. The WMT 2018 Shared Task on Parallel Corpus Filtering aimed to address this by evaluating methods for cleaning these noisy parallel corpora.

Methods

Vicomtech's approach builds upon the STACC (Set-Theoretic Alignment of Comparable Corpora) system, known for its efficiency and portability in identifying parallel sentences. For the WMT 2018 task, they extended STACC with two key modifications:

  • OOV Density Penalty: To combat noise in web-crawled data, they introduced a penalty based on the proportion of unknown words (OOV) in sentence pairs. Sentences with a low OOV density (indicating they are likely to be covered by existing vocabulary) receive a score closer to the original STACC score, while sentences with high OOV density (potentially noisy or covering new vocabulary) have their scores penalized, pushing them towards zero. This helps to down-weight potentially erroneous sentence pairs.
  • N-gram Saturation: They experimented with a method to account for data redundancy. This involves sorting sentence pairs by their STACC.OOV scores and then processing them to identify "new" n-grams (sequences of words) not previously encountered. Sentence pairs that contribute a large proportion of new n-grams are favored, aiming to demote redundant data and promote sentences that introduce novel linguistic patterns.

Crucially, their approach requires no prior training and relies on pre-trained IBM2 lexical translation tables, which can be efficiently generated.

Datasets

  • Training Data for Lexical Translation Tables: Corpora made available for the WMT 2018 news translation task were used. This included Europarl v7, Common Crawl, NewsCommentary, and the Rapid corpus of EU press releases. The Paracrawl corpus was excluded to ensure reliable lexical translation tables from less noisy bilingual corpora. This resulted in approximately 5.6 million parallel sentences for training.
  • Evaluation Data: The WMT 2018 shared task test sets were used for evaluation.

Key Insights and Ideas

  • Simplicity as a Strength: The paper highlights that a straightforward approach, like STACC with an OOV penalty, can achieve competitive results in a complex task like parallel corpus filtering, even on massive datasets.
  • OOV as a Noise Indicator: The OOV density penalty effectively leverages the idea that sentences with a high proportion of unknown words in a noisy corpus are more likely to be misaligned or contain irrelevant content.
  • N-gram Saturation's Mixed Impact: While the n-gram saturation method aimed to address data redundancy, its effectiveness was found to be mixed, performing worse for NMT systems and increasing computational cost. This suggests that simply adding new n-grams might not always be beneficial, especially for advanced models like NMT.
  • Efficiency is Key: The proposed method is highly efficient, capable of processing over 100 million sentence pairs in under an hour on a multi-core server, making it practical for real-world applications.

Key Results

  • Vicomtech's primary submission, STACC.OOV, performed well, ranking in the top third for SMT 10M and NMT 10M, and as a mid-performing system in other scenarios.
  • The system achieved competitive results overall, ranking in the top half among competing systems.
  • On average, their primary submission was 1.46 BLEU points below the best system and 2.46 points above the mean.
  • The STACC.OOV.NGSAT variant performed similarly to the primary submission for SMT but worse for NMT, and was more computationally expensive.
  • The system processed 104 million sentence pairs in under an hour with minimal RAM consumption (11.3GB for STACC.OOV).

Limitations and Future Work

  • N-gram Saturation for NMT: The paper acknowledges that the n-gram saturation method performed worse for NMT and was more resource-intensive. Further research is needed to understand why this happened and how to better account for data redundancy in NMT. One hypothesis is that NMT, with its reliance on word embeddings and contextual variants, might be negatively impacted by the imbalance created by n-gram saturation, whereas SMT might benefit more.
  • Domain Specificity: The authors note that their scoring is based purely on alignment and might not be geared towards selecting sentence pairs most informative for specific domains (e.g., news). This could explain why their approach performed exceptionally well on the KDE test set but less so on others.
  • Future Work: The authors plan to evaluate variants of their approach, particularly exploring how data features in different domains impact filtering. They also suggest that a more detailed analysis is needed to understand the negative impact of n-gram saturation on NMT.

Summary updated

BibTeX
@inproceedings{azpeitia2018stacc,
  author = {Andoni Azpeitia and Thierry Etchegoyhen and Eva Martínez Garcia},
  title = {{STACC, OOV Density and N-gram Saturation: Vicomtech’s Participation in the WMT 2018 Shared Task on Parallel Corpus Filtering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure