TL;DR
This paper introduces YiSi, a novel semantic machine translation evaluation metric, to filter noisy parallel corpora for better machine translation training. Their approach achieved top rankings in the WMT18 Parallel Corpus Filtering task, demonstrating the effectiveness of semantic similarity for identifying high-quality parallel data.
What Problem Does This Paper Address?
Training effective machine translation (MT) systems relies heavily on large, clean parallel corpora – pairs of sentences that are accurate translations of each other. However, automatically collected web-crawled corpora, like ParaCrawl, are notoriously noisy, containing many sentence pairs that are not true translations, are poorly translated, or are even non-linguistic. This noise significantly degrades MT model performance. The challenge is to efficiently and accurately identify the "clean" sentence pairs from these massive, noisy datasets. Traditional methods often focus on surface-level features like word overlap or simple alignment scores, which may not fully capture the semantic equivalence required for good translation.
Methods
The core of the NRC's submission is the YiSi metric, a semantic MT evaluation metric designed to assess the similarity between a machine-translated sentence and its reference. YiSi comes in two flavors:
- YiSi-1: Focuses on segmental semantic similarity, comparing n-grams (sequences of words) between the MT output and the reference. It calculates precision and recall based on weighted semantic similarity scores.
- YiSi-2: A crosslingual version that uses bilingual word embeddings to evaluate semantic similarity directly between the source and target sentences of a parallel pair, without needing an MT system to translate first.
Both YiSi-1 and YiSi-2 have "srl" variants (YiSi-1_srl, YiSi-2_srl) that incorporate semantic role labeling (SRL) information for a more principled comparison of semantic structures. The authors also experimented with combining various features (parallelism, fluency, grammaticality) using regression, but found that a simpler, feature-based approach using YiSi-1 precision was more effective.
Beyond the core scoring, their system included:
- Initial Filtering: Steps to remove obvious noise, such as replacing URLs and emails with placeholders, and filtering out exact duplicates or pairs where the source and target were identical. They also filtered out sentences with a high proportion of non-matching numbers or punctuation. Language identification was used to remove pairs where one sentence was likely not in the expected language.
- Redundancy Filtering: A post-scoring step to remove semantically similar sentence pairs that might lead to a limited vocabulary coverage in the final training data. This was achieved by filtering out pairs that didn't introduce new source-language word bigrams.
Datasets
- WMT18 Parallel Corpus Filtering Task Data: This included a large, noisy "dirty" ParaCrawl corpus (104 million German-English sentence pairs) and "clean" German-English monolingual and parallel corpora for training MT systems and evaluation.
- WMT18 News Translation Shared Task Data: Used for training the MT systems within the YiSi metric itself.
- Synthetic Noisy Data: The authors generated synthetic noisy data by introducing various errors (offsetting/deleting tokens, wrong language tokens, misalignments) into clean development sets (based on WMT09-13 test sets) to tune feature combinations.
Key Insights and Ideas
- Semantic Similarity is Key: The paper strongly suggests that semantic textual similarity, as captured by YiSi, is a more crucial factor for identifying good parallel data than many other features, including fluency and grammaticality, when aiming for high-quality MT training data.
- Simplicity Can Trump Complexity: Despite exploring complex feature combinations and regression models, the best performance was achieved with a simpler, single-feature approach (YiSi-1 precision), highlighting the power of a well-chosen, effective feature.
- Redundancy is a Real Problem: The redundancy filtering step, while not always impactful on large datasets, showed significant improvements on smaller subsets, indicating that simply having parallel sentences isn't enough; diversity in the training data is also important.
- Crosslingual vs. Monolingual Evaluation: YiSi-2, the crosslingual metric, showed promise but YiSi-1 (which uses a monolingual MT system internally) ultimately performed better in their experiments, suggesting that leveraging existing MT technology can be beneficial.
Key Results
The NRC's supervised submissions, particularly NRC-yisi-bicov, performed very well in the WMT18 Parallel Corpus Filtering task:
- 4th place in the 100-million-word evaluation.
- 8th place in the 10-million-word evaluation.
- 6th place overall out of 48 submissions.
- It was one of only four submissions ranked in the top 10 for both the 10M and 100M word evaluations.
Internal MT quality checks showed that data filtered by YiSi-1 precision led to better MT performance (higher BLEU scores) compared to random baselines or other features. The redundancy filtering step provided substantial BLEU score increases (+1.61 for SMT, +2.44 for NMT) on the smaller 10M-word selections.
Limitations and Future Work
- Feature Combination Challenges: The authors acknowledge that combining YiSi with other fluency and grammaticality features proved difficult and did not yield better results than using YiSi alone. Their attempt to synthesize noisy data for tuning feature combinations was not as successful as anticipated.
- 10M-word Performance: While strong overall, their systems showed relatively weaker MT performance on the 10M-word condition compared to the 100M-word condition. This suggests that fluency and grammaticality features, while not dominant, are still useful for fine-grained ranking of MT training data quality.
- Future Work: The authors suggest investigating more realistic tuning example generation and exploring other feature combination methodologies to better leverage the strengths of various features. They also note the potential for further improvements by incorporating more sophisticated fluency and grammaticality features.
Summary updated