PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

Yinfei Yang, Yuan Zhang, Chris Tar, Jason Baldridge

Abstract

Yinfei Yang, Yuan Zhang, Chris Tar, Jason Baldridge. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Paraphrase identification is a fundamental task in natural language processing (NLP), yet many existing models rely heavily on superficial word overlap rather than deep semantic understanding. PAWS-X is a new multilingual benchmark designed to push models beyond these "lazy" heuristics. By extending the English Paraphrase Adversaries from Word Scrambling (PAWS) dataset to six additional languages, this research highlights the challenges of capturing structural and contextual nuances across diverse linguistic families.

The Challenge of Lexical Overlap

Traditional paraphrase datasets often contain pairs that are either very similar in wording and meaning, or very different in both. This allows models to achieve high accuracy simply by measuring lexical overlap. However, adversarial pairs—sentences with high word overlap but different meanings (e.g., "The dog bit the man" vs. "The man bit the dog")—expose the limitations of these models. While the original PAWS dataset addressed this for English, the lack of high-quality multilingual adversarial data has hindered the evaluation of cross-lingual models used in global search and translation tools.

Dataset Construction

The researchers created PAWS-X by expanding the Wikipedia portion of the original PAWS dataset into French (fr), Spanish (es), German (de), Chinese (zh), Japanese (ja), and Korean (ko).

Example of an adversarial pair from PAWS-X across multiple languages.

The construction involved a hybrid approach:

  1. Human Translation: Professional translators handled the development and test sets (~4,000 pairs per language) to ensure a high-quality "gold standard" for evaluation.
  2. Machine Translation (MT): To scale the training data, a neural machine translation service was used to translate approximately 49,000 pairs per language.

Experimental Setup

The study evaluated three distinct model architectures to establish benchmarks:

  • Bag-of-Words (BOW): A baseline that ignores word order entirely.
  • ESIM: A BiLSTM-based model designed to capture local sequence information.
  • Multilingual BERT (mBERT): A state-of-the-art Transformer-based model pre-trained on 104 languages.

Models were tested using several strategies, most notably Zero-Shot (training on English only) and Merged (training on a combination of English and machine-translated target language data).

Results and Analysis

The results demonstrate that PAWS-X is a significantly more difficult benchmark than standard paraphrase datasets, successfully exposing the reliance on word overlap.

Performance of BOW, ESIM, and BERT models across seven languages.

Key Findings:

  • mBERT Dominance: mBERT outperformed other models, achieving an average accuracy of 87.2% across non-English languages using the "Merged" strategy.
  • The "Merged" Advantage: Combining original English data with machine-translated data yielded the best results, suggesting that even noisy MT data provides a useful signal for cross-lingual transfer.
  • Structural Bottlenecks: The massive 32% accuracy gap between mBERT and BOW confirms that PAWS-X successfully measures a model's ability to process syntax and word order rather than just vocabulary.
  • The Language Gap: Models performed better on Indo-European languages (French, Spanish, German) than on CJK languages (Chinese, Japanese, Korean). This is likely due to both MT quality and the greater syntactic distance of CJK languages from the English source.

Discussion and Future Work

While mBERT shows strong cross-lingual capabilities, there remains "considerable headroom" for improvement, particularly in CJK languages where accuracy lagged around 83-84%. One specific challenge identified was "entity noise," where inconsistent translations of named entities (e.g., "Slovak" vs. "Slowake") can confuse models. Future research should focus on improving non-local context capture and ensuring entity consistency across translations to further close the performance gap in adversarial multilingual settings.

Summary updated

BibTeX
@inproceedings{yang2019paws,
  author = {Yinfei Yang and Yuan Zhang and Chris Tar and Jason Baldridge},
  title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=YEN86PWGQnA},
}
Figure