TL;DR
This paper introduces MORPHEUS-MULTILINGUAL, a novel method to systematically test the robustness of neural machine translation (NMT) systems when faced with grammatical errors in the source text, specifically focusing on morphological inflections. They demonstrate that while NMT systems trained on large datasets are generally robust, the degree of robustness can vary significantly across languages and is not directly correlated with morphological richness.
What Problem Does This Paper Address?
Neural Machine Translation (NMT) systems have achieved remarkable success, but they often struggle when presented with noisy or ungrammatical input. This is particularly problematic when translating text from non-native speakers, who frequently make morphological errors (e.g., incorrect verb conjugations, noun plurals). While previous work has explored this issue for English-to-X translation, there's a significant gap in understanding how X-to-English systems, especially those dealing with morphologically rich languages, handle such input variations. This paper aims to quantify and analyze the robustness of X-to-English NMT systems against morphological perturbations.
Methods
The core of this work is the MORPHEUS-MULTILINGUAL toolkit. It builds upon the existing MORPHEUS toolkit but extends it to handle X-to-English translation. The methodology involves:
- Reinflection: Using the UniMorph project's dictionaries and the
unimorph_inflecttool, the system generates various inflectional forms for nouns, adjectives, and verbs in the source text. - Adversarial Perturbation: For each input sentence, the tool systematically iterates through tokens, identifying inflectional edits that cause the largest drop in translation quality (measured by chrF score, which is more sensitive to morphological variations than BLEU). They generate a pool of perturbed sentences and select the one that maximally degrades performance.
- Evaluation: The perturbed sentences are then used to evaluate NMT systems. They also introduce a "Target-Source Noise Ratio" (NR) metric to better understand robustness, where an NR of ~1 indicates the system is proportionally robust to changes.
Datasets
- Multilingual TED Corpus: Used to train transformer-based NMT systems from scratch for 11 language pairs.
- WMT19 News Translation Shared Task Data: Used to evaluate pre-trained NMT models (fairseq) for Russian→English and German→English.
- German Falko-MERLIN-GEC and Russian RULEC-GEC Datasets: Used to evaluate robustness on actual learner text, focusing on morphological errors.
Key Insights and Ideas
- X→English is Different: The challenges of translating from morphologically rich languages to English are distinct from English-to-X translation, as word order and inflectional changes can have a more significant impact on meaning.
- Training Data is Key: The study strongly suggests that the size of the parallel training corpus is the most significant factor in NMT robustness against morphological perturbations. Larger datasets lead to more robust models.
- Morphological Richness is Misleading: Contrary to intuition, the morphological richness of a language (as measured by Type-Token Ratio) does not directly correlate with the robustness of its NMT system. High-resource languages like Russian and Turkish, despite being morphologically rich, show varying degrees of robustness.
- chrF is Better for Morphological Errors: The authors found chrF (character n-gram based metric) to be more reliable than BLEU for evaluating the impact of morphological perturbations, as it better captures variations in word forms.
- Semantic Shifts Can Occur: Some aggressive perturbations can lead to significant semantic changes in the translation, highlighting the complexity of handling morphological errors.
Key Results
- Performance Drops: For the multilingual TED corpus experiments, adversarial perturbations led to significant drops in chrF scores, ranging from 0% to 54% across different language pairs.
- Pre-trained Models: State-of-the-art pre-trained models from fairseq also showed noticeable performance degradation on adversarial sets for Russian→English and German→English.
- Learner Text Robustness: Pre-trained NMT models demonstrated surprising robustness to morphological errors found in actual learner text from GEC datasets.
- Noise Ratio (NR): While BLEU/chrF scores dropped, the NR metric often remained close to 1 for high-resource languages, indicating that the systems were proportionally adapting to the changes, even if the absolute scores decreased.
Limitations and Future Work
- Synthetic vs. Real Errors: The adversarial sets are synthetic. While they provide a controlled way to test robustness, they may not perfectly capture the nuances of real-world learner errors.
- Aggressive Edits: In morphologically rich languages like Russian, the algorithm can generate highly aggressive edits that result in unrealistic sentences, deviating from the intended simulation of learner text.
- Semantic Change Handling: The paper acknowledges that some perturbations can cause significant semantic shifts, and handling these requires a deeper understanding of language-specific grammar, which is left for future work.
- Adversarial Training: While preliminary experiments with adversarial training were conducted, they showed marginal improvements and require further extensive evaluation, especially on models with already high BLEU scores.
- Ensemble Models: Due to resource constraints, the study did not evaluate ensemble NMT models, which might exhibit different robustness characteristics.
Summary updated