El Departamento de Nosotros: How Machine Translated Corpora Affects Language Models in MRC Tasks

Maria Khvalchik, Mikhail GalkinarXiv 2020 · arXiv

Abstract

Machine translation (MT) is a common shortcut used to bootstrap datasets for non-English languages in Natural Language Processing. However, this study demonstrates that raw MT introduces subtle artifacts that can significantly degrade the performance and robustness of Language Models (LMs) in Machine Reading Comprehension (MRC) tasks. By comparing raw MT with a curated "Translate-Align-Retrieve" (TAR) approach, the authors show that data quality is paramount. Human evaluations and experimental benchmarks on Spanish MRC tasks reveal that models trained on curated data are more robust and perform better on external "gold standard" benchmarks like XQuAD and MLQA.

Introduction: The "Data Wall" and the MT Shortcut

Most state-of-the-art NLP models are pre-trained on massive English corpora. Applying these models to other languages, such as Spanish, often encounters a "data wall" due to the lack of high-quality, human-annotated datasets like SQuAD. To circumvent this, researchers often use Neural Machine Translation (NMT) to translate English datasets. However, NMT often struggles with gender inference, named entities, and word order, leading to errors like the "Department of Us" (a literal translation of "US War Department" as Departamento de Guerra de Nosotros). This paper investigates whether modern transformers like BERT can overlook these translation artifacts or if they fundamentally mislead the models.

Human Evaluation of Translation Quality

The authors conducted a user study where 12 Spanish speakers evaluated 50 parallel examples from two Spanish versions of SQuAD 1.1: a raw MT version and a curated TAR version. The TAR pipeline includes post-processing to ensure that answer spans in the text align correctly with translated questions. Evaluators scored examples on a scale of 0 (unintelligible) to 2 (minor or no mistakes).

Distribution of human evaluation scores (0-2) for TAR and MT datasets.

The human evaluation clearly favored the TAR approach. As shown in the distribution of scores, TAR-processed data received significantly more "2" ratings compared to the raw MT data, which frequently resulted in unintelligible or highly flawed translations. Quantitatively, TAR data achieved an average score of 1.717/2.0, while raw MT lagged behind at 1.320/2.0.

Experimental Benchmarking and Results

The study fine-tuned several models—multilingual BERT (mBERT), a Spanish-specific BERT (BETO), and DistilledBETO—on both MT and TAR datasets. These models were then cross-evaluated on internal dev sets and external benchmarks like XQuAD and MLQA.

Performance of various models across different versions of the es-SQuAD dataset, highlighting the gap between TAR and MT training.

Key Insights

  • Artificial Overperformance: Models trained on raw MT data often show high scores when tested on other raw MT data. However, their performance craters when tested on human-curated data. This suggests the models are learning the specific "dialect" of the translation engine rather than the underlying language.
  • The Performance Gap: On the TAR dev set, models trained on TAR data outperformed MT-trained versions by massive margins. For instance, mBERT trained on MT SQuAD 2.0 scored roughly 32 points lower in Exact Match (EM) and F1 compared to the Spanish-specific BETO model.
  • Multilingual Resilience: Multilingual models like mBERT showed more resilience to translation artifacts than monolingual models, likely due to their exposure to diverse linguistic structures during pre-training.
  • The Superiority of BETO: On external benchmarks, the Spanish-specific BETO model emerged as the clear winner, proving that language-specific pre-training combined with high-quality data is the gold standard for MRC tasks.

Conclusion

The quality of translated training sets is critical for the final performance of MRC models. The TAR method, which involves careful alignment and post-processing, consistently produces more robust models that generalize better to human-verified benchmarks. While Spanish is a relatively high-resource language, the impact of translation artifacts is likely even more severe for low-resource languages. Future research should focus on making neural architectures inherently more robust to noisy, artificially created data to reduce the need for expensive human intervention.

Summary updated

BibTeX
@misc{khvalchik2020el,
  author = {Maria Khvalchik and Mikhail Galkin},
  title = {{El Departamento de Nosotros: How Machine Translated Corpora Affects Language Models in MRC Tasks}},
  year = {2020},
  eprint = {2007.01955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.01955},
}
Figure