Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap

Qiwei Peng, David Weir, Julie WeedsACL 2023

TL;DR

This paper argues that standard paraphrase detection benchmarks are insufficient for real-world applications like paraphrase retrieval, which require models to understand nuanced degrees of semantic overlap. The authors introduce novel adversarial datasets and demonstrate that current state-of-the-art paraphrase models and sentence encoders struggle to distinguish between exact and partial paraphrases, often being misled by lexical similarity.

What Problem Does This Paper Address?

Paraphrase detection, the task of identifying sentences with similar meanings, is crucial for many Natural Language Understanding (NLU) applications like question answering and fact-checking. Traditionally, this problem is framed as a binary classification task: is this pair a paraphrase or not? However, the authors contend that this setup is inadequate for many practical scenarios, particularly paraphrase retrieval. In retrieval, the goal isn't just to find any paraphrase, but to find the closest paraphrases of a target sentence from a collection. This requires models to be sensitive to the degree of semantic overlap, not just a binary "yes" or "no." Existing datasets often fail to test this sensitivity, leading to models that perform well on standard benchmarks but falter when faced with the subtle distinctions required for retrieval.

Methods

The core of the paper's contribution lies in its novel evaluation paradigms. To address the limitations of existing datasets, the authors propose two main approaches:

  1. Generation-based Adversarial Evaluation: They use a GPT-2 generation model to create partial paraphrases by adding text to existing positive sentence pairs from standard datasets. This introduces additional information, diluting the semantic overlap and creating a scenario where the original sentence pair is an exact paraphrase, while the modified one is a partial paraphrase.
  2. Swap-based Adversarial Construction: Inspired by previous work on adversarial attacks, they develop a "multi-swap" strategy. This involves tagging words and phrases in a sentence, grouping them by part-of-speech or named entity tags, and then swapping the positions of words/phrases within these groups. This method aims to create partial paraphrases that maintain high lexical overlap and sentence length, making it harder for models to rely solely on surface-level cues. They systematically create sets of sentence pairs with decreasing degrees of semantic overlap.

Datasets

The authors leverage existing datasets for training and initial evaluation, and then create new adversarial evaluation datasets:

  • Training/Initial Evaluation: PAWSWiki and PAWSQQP (Paraphrase Adversaries from Word Scrambling) are used to fine-tune models. These datasets are known for containing high-quality sentence pairs, mostly exact paraphrases, generated through word scrambling and back-translation.
  • New Evaluation Datasets:
    • Generation-based: Created by augmenting positive pairs from PAWS test sets with GPT-2 generated text.
  • Swap-based: Created using the multi-swap strategy on positive pairs from PAWS test sets, resulting in ranked sets of paraphrases with decreasing semantic overlap.

Key Insights and Ideas

  • Binary Classification is Insufficient: The paper strongly suggests that the traditional binary paraphrase classification task doesn't capture the nuances needed for real-world applications like retrieval.
  • Lexical Overlap is a Red Herring: Models, especially sentence encoders, are heavily influenced by lexical overlap. High lexical similarity can lead to high scores even when semantic overlap is not complete, masking a lack of true semantic understanding.
  • Partial Paraphrases are a Major Challenge: Even state-of-the-art models like BERT and RoBERTa, which perform exceptionally well on exact paraphrase detection, struggle significantly when faced with partial paraphrases, especially those with subtle semantic differences.
  • Ranking is a Better Evaluation Metric: Evaluating models in a ranking scenario (where they must order sentence pairs by semantic overlap) provides a more robust assessment of their ability to discern degrees of paraphrase than simple classification.

Key Results

  • Classification Performance Drop: When tested on the generation-based adversarial datasets, BERT and RoBERTa showed a dramatic drop in accuracy after just one word was added to a sentence, indicating a lack of sensitivity to partial paraphrases.
  • Ranking Performance: In the swap-based ranking evaluation, sentence encoders (SimCSE, SBERT) showed limited ability to distinguish exact from partial paraphrases, evidenced by low R-Precision scores. While paraphrase models (BERT, RoBERTa) performed better in ranking, their scores for partial paraphrases were often higher than for exact paraphrases, suggesting their confidence scores are not reliable indicators of semantic closeness.
  • Impact of Lexical Overlap: Reducing lexical overlap through back-translation significantly degraded the performance of all evaluated models, particularly sentence encoders, reinforcing the idea that they rely heavily on word matching.

Limitations and Future Work

  • Model Scope: The study primarily focused on BERT-base and RoBERTa-base models. The authors suggest that larger language models could be considered.
  • Adversarial Data Quality: The swap-based strategy sometimes produced semantically problematic but grammatically correct sentences. Future work could involve using naturally occurring sentences and reducing reliance on irregular word or phrase usages.
  • Dataset Size: The authors acknowledge that the swap-based strategy resulted in relatively small evaluation datasets and suggest producing larger ranking datasets.
  • Future Directions: The paper calls for more research into improving models' sensitivity to different degrees of semantic overlap and exploring specific ranking objectives that better capture semantic proximity.

Summary updated

BibTeX
@inproceedings{peng2023testing,
  author = {Qiwei Peng and David Weir and Julie Weeds},
  title = {{Testing Paraphrase Models on Recognising Sentence Pairs at Different Degrees of Semantic Overlap}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure