Abstract
Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 human-translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacities to capture non-local context and sentence structure. Multilingual BERT (mBERT) fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8% accuracy across the non-English languages. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge for multilingual research.
Introduction: The Challenge of Word Overlap
In Natural Language Processing (NLP), paraphrase identification—determining if two sentences share the same meaning—is a fundamental task. However, many models "cheat" by relying on word overlap. If two sentences share 90% of the same words, a model might label them a paraphrase, even if the meaning is reversed (e.g., "The dog bit the man" vs. "The man bit the dog").
The original PAWS dataset addressed this for English by providing "adversarial" examples: pairs with high word overlap but different meanings. PAWS-X extends this challenge to a multilingual context, allowing researchers to measure how well models handle structural reasoning in French, Spanish, German, Chinese, Japanese, and Korean.
Dataset Generation
The researchers expanded the Wikipedia portion of the original English PAWS dataset using a hybrid approach:
- Human Translation for Evaluation: To ensure a reliable benchmark, professional translators converted the development and test sets (approximately 4,000 pairs per language) into the six target languages.
- Machine Translation (MT) for Training: Given the scale of the training set (nearly 50,000 pairs), Google’s Neural Machine Translation was used to generate training data for the target languages.
Experimental Setup
Three model architectures were evaluated to determine their sensitivity to sentence structure:
- Bag-of-Words (BOW): A baseline that ignores word order entirely.
- ESIM: A neural network (BiLSTM) that captures sequence information.
- Multilingual BERT (mBERT): A transformer model pre-trained on 104 languages, capable of capturing deep contextual interactions.
Training strategies included Zero-Shot (training only on English), Translate-Train (training on machine-translated target language data), and Merged (training on all available data).
Key Results and Insights
mBERT Dominance
Multilingual BERT significantly outperformed other models, achieving between 83.1% and 90.8% accuracy across non-English languages. It showed an average accuracy gain of 23% over the next best model (ESIM), demonstrating that deep cross-lingual pre-training is essential for transferring structural understanding between languages.
The "Merged" Strategy
The best performance consistently came from the "Merged" strategy—training on both the original English data and the machine-translated data for all other languages. This suggests that even noisy machine-translated data provides valuable signal for learning language-specific structural nuances.
Typological Difficulty
Models consistently performed worse on Chinese, Japanese, and Korean (CJK) compared to Indo-European languages like German or French. This performance gap is likely due to the greater typological distance from English and lower machine translation quality for these language pairs.
Conclusion
PAWS-X establishes a rigorous new benchmark for multilingual NLP. While mBERT shows impressive results, the remaining "headroom" between current performance and perfect accuracy indicates that PAWS-X will continue to challenge models to move beyond simple keyword matching toward true structural and contextual understanding. Future work may focus on improving cross-lingual transfer for CJK languages and refining models to be more sensitive to subtle syntactic shifts.
Summary updated