SYSTRAN Participation to the WMT2018 Shared Task on Parallel Corpus Filtering

MinhQuang Pham, Josep Crego, Jean SenellartEMNLP 2018

TL;DR

This paper presents SYSTRAN's approach to the WMT2018 parallel corpus filtering task, introducing a neural sentence similarity classifier designed to identify high-quality parallel sentence pairs from noisy web-crawled data. Their method leverages deep learning to predict sentence relatedness, aiming to improve machine translation performance by cleaning training corpora without relying on extensive domain-specific feature engineering.

What Problem Does This Paper Address?

High-quality parallel corpora are crucial for training effective machine translation (MT) systems, especially neural MT (NMT) models that require vast amounts of data. However, automatically collected parallel corpora, like those from web crawling, are often noisy. This noise can manifest as incorrect language pairs, sentences that are not translations of each other, or poorly formed sentences. The WMT2018 shared task on parallel corpus filtering aimed to address this by challenging participants to develop methods for cleaning these noisy corpora, producing smaller subsets of high-quality sentence pairs. The goal is to improve MT performance by filtering out "bad" pairs, not necessarily by selecting pairs relevant to a specific domain.

Methods

SYSTRAN's approach centers on a neural sentence similarity classifier. This model is inspired by work on word alignment and uses a Bi-directional LSTM architecture to encode both source and target sentences. For each sentence, the Bi-LSTM outputs forward and backward hidden states, which are then concatenated to form a single vector representing each word in context. The final hidden states of the Bi-LSTM are further combined to create a vector representing the entire sentence.

The core of the similarity measurement is the cosine similarity between the sentence vectors. To enable unsupervised training, the model uses an aggregation operation to combine word-level alignment scores. A soft-margin objective is employed to increase scores for true target words and decrease scores for non-present words. The model is trained using a loss function that aims to maximize the similarity score for actual parallel sentence pairs while minimizing it for negative examples.

Negative examples are generated in several ways:

  • Random unpaired sentences: Labeled as divergent.
  • Word replacement: Random sequences of words are replaced with others of the same part-of-speech to create less obvious divergences.
  • Sentence insertion: A second sentence is inserted at the beginning or end of a pair to simulate segmentation errors.

To avoid trivial predictions based on length differences, negative examples are constrained to have length ratios not exceeding 2.0 (or 3.0 for very short sentences).

Datasets

The primary dataset used is the 1 billion word German-English corpus crawled from the web as part of the Paracrawl project, provided for the WMT2018 shared task. This corpus is described as extremely noisy.

For evaluation, participants had to create subsets of 10 million and 100 million words. MT systems were then trained on these subsets and evaluated using the BLEU score on six blind test sets, including the official WMT 2018 news translation test set and another undisclosed test set.

Key Insights and Ideas

  • Deep Learning for Filtering: The paper demonstrates the effectiveness of using deep neural networks, specifically Bi-LSTMs, for the task of parallel corpus filtering, moving beyond traditional rule-based or feature-engineered methods.
  • Unsupervised Training: The ability to train the similarity classifier using unsupervised methods (i.e., without explicit human-annotated parallel/non-parallel sentence pairs for the training data itself) is a significant advantage for large-scale corpus cleaning.
  • Flexibility of Neural Models: The neural approach offers flexibility, as it doesn't require domain-specific feature engineering, making it adaptable to different language pairs and corpus types.
  • Importance of Negative Examples: The careful construction of diverse negative examples (random, word-replacement, sentence-insertion) is crucial for training a robust classifier that can distinguish subtle divergences.

Key Results

The paper reports that SYSTRAN's submission achieved competitive results in the WMT2018 shared task. While specific ranking details are not elaborated upon in the provided text, Figure 2 shows that SYSTRAN's system (labeled "Systran") performed comparably to many other participants, with BLEU scores for the 100 million token corpus falling within a 3-point margin of the top performers. The authors state they succeeded in their objective of building a simple network capable of filtering out divergent sentence pairs, assisted by basic length and language identification rules.

Limitations and Future Work

  • Simplicity of Filtering: The authors acknowledge that their approach, while effective, was assisted by very simple filtering techniques based on length and language identification. The neural similarity classifier itself is described as a "very simple network."
  • Performance on Subtle Divergences: While the method aims to predict relatedness, the paper doesn't deeply analyze its performance on extremely subtle divergences or cases where sentences might be grammatically correct but not translations.
  • Future Work: Not explicitly discussed in the provided sections. However, the success of this neural approach suggests potential for further research into more sophisticated neural architectures, attention mechanisms, or multi-task learning for improved filtering accuracy and robustness. Exploring different aggregation strategies or loss functions could also be avenues for future work.

Summary updated

BibTeX
@inproceedings{pham2018systran,
  author = {MinhQuang Pham and Josep Crego and Jean Senellart},
  title = {{SYSTRAN Participation to the WMT2018 Shared Task on Parallel Corpus Filtering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure