TL;DR
This paper proposes a systematic approach to generating high-quality NLP corpora for non-English languages using machine translation (MT) and automatic quality assessment metrics. By leveraging back-translation and carefully filtering the results, they demonstrate that it's possible to create valuable datasets for tasks like Reading Comprehension and Word Sense Disambiguation, even without extensive human annotation.
What Problem Does This Paper Address?
The advancement of Natural Language Processing (NLP) has been heavily reliant on the availability of large, high-quality datasets, particularly for English. However, creating such corpora for the vast majority of the world's languages is prohibitively expensive and time-consuming, especially when expert linguistic knowledge is required for annotation. This creates a significant barrier to developing robust NLP tools for non-English speakers. While machine translation (MT) offers a potential solution for generating data, ensuring the quality of these automatically translated corpora is a major challenge. Existing methods often rely on simple metrics like BLEU, which have known limitations in accurately reflecting human judgment of translation quality.
Methods
The core of this paper's methodology is a back-translation strategy combined with automatic quality assessment.
- Back-Translation: The process starts with an existing English NLP corpus for a specific task. This corpus is translated into the target non-English language using an MT system. Then, this translated corpus is translated back into English. The idea is that if the back-translated text closely matches the original English text, it suggests that the intermediate translation into the target language likely preserved the original meaning.
- Automatic Quality Assessment: Instead of relying solely on BLEU, the authors explore other metrics to assess the quality of the back-translations. They compare a combination of BLEU and ROUGE (termed FB+R) with the more advanced Meteor Universal (M.U.) metric. The hypothesis is that a metric with higher correlation to human assessment will lead to better corpus selection.
- Corpus Filtering: The key innovation lies in how they use these metrics to filter the generated corpus. They investigate different thresholds for the quality scores, aiming to identify a sweet spot that balances corpus size with translation quality. This involves ranking training samples based on the chosen metrics and experimenting with different percentages (e.g., top 20% to 70%) to see how task performance is affected.
- Task-Specific Processing: For Reading Comprehension, they specifically filter out samples where the answer is no longer explicitly present in the back-translated reference text. For Word Sense Disambiguation, they focus on sentences with one-to-one word alignments for ambiguous words.
Datasets
- Reading Comprehension (Spanish): The authors used the English SQuAD 1.1 dataset as their source. They translated the training and development subsets into Spanish and then back-translated them. A new test set was generated from these subsets.
- Word Sense Disambiguation (Portuguese): For this task, they utilized the Unified Evaluation Framework (UEF) for English WSD, which is aligned with WordNet. They partially translated this into Portuguese and then filtered it. They also used the native Portuguese CSTNews corpus as a curated test set.
Key Insights and Ideas
- Back-translation is a viable strategy: The paper strongly suggests that back-translation, when combined with appropriate filtering, is a feasible method for generating useful NLP corpora for non-English languages.
- Beyond BLEU: The study highlights the limitations of BLEU and demonstrates that more sophisticated metrics like Meteor Universal can lead to better corpus selection and, consequently, improved task performance.
- Quality over Quantity (to an extent): The experiments suggest that carefully selecting a high-quality subset of translations, even if smaller, can be more beneficial than using a larger, noisier corpus.
- Systematization is possible: The authors aim to provide a generalizable framework that can be applied to various tasks and languages, reducing the need for language-specific expertise in corpus creation.
Key Results
- Reading Comprehension (Spanish): When evaluating their generated corpus using a neural network model, they found that their method, trained on a corpus filtered by quality metrics (specifically, the top 40% of the training set using M.U. score), outperformed a non-data-driven pipeline method (Vicedo et al., 2004).
- Word Sense Disambiguation (Portuguese): For the Lexical Sample task, their generated corpus, filtered using the FB+R criterion at the 40% partition, achieved better precision than both the Most Frequent Sense (MFS) heuristic and an adaptation of the Lesk algorithm. While not outperforming these baselines on all verbs, the overall improvement is significant.
Limitations and Future Work
- MT System Dependency: The quality of the generated corpus is inherently dependent on the quality of the underlying MT system. The paper acknowledges that using less robust MT systems could be a future research direction to assess the limits of this strategy.
- Generalizability: While the goal is generalization, the study focused on two specific tasks and languages. Further work is needed to explore a wider range of tasks and languages, especially low-resource ones.
- Metric Correlation: While Meteor Universal showed promise, the paper suggests further exploration of automatic evaluation metrics and their correlation with human judgment.
- Complementary Techniques: The authors propose complementing back-translation with cross-lingual embeddings or other supervised/unsupervised methods to further enhance corpus quality.
- Extensive Exploration: A more exhaustive exploration of different filtering thresholds, metrics, and languages is planned.
Summary updated