TL;DR
This paper introduces "dual conditional cross-entropy filtering," a novel method for cleaning noisy parallel corpora used in machine translation. By leveraging two inverse translation models, their approach effectively identifies and removes harmful sentence pairs, leading to significantly improved translation quality and achieving top rankings in the WMT 2018 shared task on parallel corpus filtering.
What Problem Does This Paper Address?
The explosion of readily available web-crawled parallel corpora, like Paracrawl, has been a boon for machine translation research. However, these corpora are notoriously noisy, containing misaligned sentences, incorrect translations, and even non-translation pairs. Neural machine translation (NMT) models, in particular, are highly sensitive to this noise, leading to degraded performance. Existing data selection methods often focus on monolingual data or simple filtering techniques, leaving a gap in effectively cleaning bilingual data for NMT. This paper addresses the critical need for a robust method to filter out these harmful sentence pairs from large, noisy parallel corpora to improve NMT model quality.
Methods
The core contribution is dual conditional cross-entropy filtering. This method operates on sentence pairs (x, y) from a noisy parallel corpus and uses two inverse translation models: one translating from source to target (A: x -> y) and another from target to source (B: y -> x). These models are trained on a clean, high-quality parallel corpus.
The filtering process involves two main components:
-
Adequacy Score: This score measures the agreement between the two inverse translation models. It's calculated as the absolute difference between the conditional cross-entropies of the models for the given sentence pair, normalized by the sum of their cross-entropies. A low difference indicates high agreement, suggesting the pair is likely a good translation. The formula is: $adq(x, y) = exp(-(|H_A(y|x) - H_B(x|y)| + \frac{1}{2} (H_A(y|x) + H_B(x|y))))$.
-
Language Identification Filter: A simple but effective filter that ensures both sentences in a pair are identified as the desired languages (e.g., German and English).
The paper also explores a cross-entropy difference filter (inspired by Moore and Lewis) for monolingual data, which uses an in-domain model and an out-of-domain model to identify sentences that are more likely to belong to the target domain.
Datasets
- Paracrawl Corpus: The primary noisy parallel corpus used for filtering and training.
- WMT Parallel Data: Used to train the clean "helper" models (translation and language models) for filtering. This data excludes Paracrawl.
- WMT2018 Shared Task Data: The organizers provided a 1-billion-word noisy German-English corpus from the Paracrawl project. Participants were asked to sub-select subsets of 100 million and 10 million words.
- WMT2016 and WMT2017 Test Sets: Used for evaluating the performance of NMT models trained on filtered data.
Key Insights and Ideas
- Bilingual Adequacy is Key: Unlike monolingual filtering, the paper emphasizes modeling bilingual adequacy by ensuring consistency across inverse translation directions. This directly addresses the core task of parallel corpus filtering.
- NMT Sensitivity to Noise: The paper highlights that NMT models are significantly more sensitive to noisy parallel data than older phrase-based SMT models, underscoring the urgency of effective filtering.
- Dual Learning Connection: The proposed filtering method shares a strong conceptual link with dual learning criteria, suggesting that data preferred by models trained on clean data can also be used to train better models.
- Simple Filters Can Be Powerful: The language identification filter, while basic, proved to be a crucial component, removing a substantial portion of irrelevant data.
- Trade-offs in Filtering: The paper demonstrates that different filtering strategies (e.g., adequacy vs. similarity) have varying impacts, and combining them can lead to optimal results.
Key Results
- Improved BLEU Scores: Models trained on data filtered with the proposed dual conditional cross-entropy method achieved higher BLEU scores compared to models trained on clean WMT data and even on filtered Paracrawl data using simpler methods.
- WMT2018 Shared Task Success: The authors' submission, using the
de-en-adq.dom0.25filter, achieved the overall highest ranking in the WMT2018 Shared Task on Parallel Corpus Filtering. They scored top in three out of four subtasks (SMT 10M, NMT 10M, and NMT 100M) and third in the SMT 10M subtask, out of 48 submissions per subtask. - Ablation Studies: Removing components of the dual conditional cross-entropy filter (e.g., the absolute difference or the CE weighting) led to a decrease in BLEU scores, indicating the usefulness of each component.
Limitations and Future Work
- Helper Model Quality: The effectiveness of the filtering method relies on the quality of the "helper" translation and language models trained on clean data. The paper notes that these models were trained on small subsamples of clean data and suggests investigating if stronger models would yield better discriminators.
- Domain Specificity: While the shared task encouraged not optimizing for a specific domain, the authors' experience suggests that clean news data is generalizable. However, further exploration of domain-specific filtering might be beneficial.
- Future Directions:
- Formalizing the method and analyzing its connection to dual learning training criteria.
- Investigating the use of stronger translation and language models for scoring.
- Exploring bilingual cross-entropy difference filtering.
- Experimenting with conditional probability distributions from translation models for domain filtering.
- Using a single multilingual model for the dual conditional cross-entropy method, rather than two inverse models, to potentially improve agreement.
Summary updated