Introduction
In the field of Natural Language Processing (NLP), data scarcity remains a significant hurdle for languages other than English. Even for widely spoken languages like Spanish, researchers often rely on machine translation (MT) to adapt English benchmarks, such as SQuAD, for local training. However, this "quick fix" introduces syntactic errors and "translationese"—patterns that do not exist in natural language. This paper investigates whether modern architectures like BERT are robust enough to ignore this noise or if the quality of translated data fundamentally limits model reliability in Machine Reading Comprehension (MRC) tasks.
Methodology: TAR vs. Raw MT
The researchers compared two distinct approaches to generating Spanish training data:
- Raw MT: Direct translation of SQuAD 1.1 and 2.0 using a private neural machine translation (NMT) provider.
- TAR (Translate-Align-Retrieve): A sophisticated methodology that employs post-processing to improve alignment between questions, answers, and context, reducing the artifacts common in raw translation.
To validate these methods, a user study was conducted where 12 Spanish speakers rated 50 parallel examples on a scale of 0 (poor) to 2 (excellent). Subsequently, the researchers fine-tuned mBERT (multilingual) and BETO (Spanish-specific BERT) on these datasets to evaluate their performance.
Human Evaluation of Translation Quality
The human evaluation revealed a stark contrast in quality between the two methods. The TAR method produced significantly more reliable translations, achieving an average score of 1.717 compared to 1.320 for raw MT.
As shown in the distribution, TAR translations received the highest score (2) in 80% of cases, whereas raw MT reached this level of quality only 50% of the time. This gap suggests that raw MT often fails to capture the nuances required for high-quality training data.
The "Overfitting to Noise" Trap
A critical finding of the study is the tendency of models to overfit to translation artifacts. The paper’s title, "Departamento de Nosotros," refers to a notable MT failure where "US War Department" was translated as "Departamento de Guerra DE NOSOTROS" (War Department of Us). The MT engine misinterpreted the abbreviation "US" as a pronoun rather than the United States.
Models fine-tuned on raw MT data performed well when tested on other MT-generated data but failed to generalize to curated sets. For instance, mBERT trained on MT data performed over 30 points worse in Exact Match (EM) than BETO when evaluated on the curated TAR dataset. This indicates that models trained on low-quality MT data learn to exploit specific translation errors rather than developing a genuine understanding of the language.
Key Results and Benchmark Performance
To test generalization, the models were evaluated on external, high-quality benchmarks: MLQA and XQuAD. These benchmarks represent "unseen" extractive QA tasks that are not influenced by the specific translation artifacts of the training sets.
Across these benchmarks, the monolingual BETO model consistently outperformed multilingual versions when fine-tuned on high-quality data. Interestingly, multilingual models (mBERT) showed slightly more resilience to MT artifacts than monolingual models on external benchmarks, likely because pre-training on diverse languages helps the model generalize over language-specific noise.
Conclusion
The study concludes that while machine translation is a necessary tool for addressing data scarcity, "raw" MT introduces significant risks. Curated datasets using the TAR method produce far more robust and reliable models. For Spanish-specific tasks, the combination of a monolingual model like BETO and high-quality curated data remains the gold standard. Future work will focus on making neural architectures more inherently "noise-resilient" to handle the inevitable flaws of machine-translated corpora without requiring extensive human intervention.
Summary updated