Improving Paraphrase Detection with the Adversarial Paraphrasing Task

Animesh Nighojkar, John LicatoACL 2021

TL;DR

This paper introduces the Adversarial Paraphrasing Task (APT) to create paraphrases that are semantically equivalent but lexically and syntactically dissimilar. This novel approach aims to train paraphrase detection models to focus on true meaning equivalence rather than superficial word overlap, leading to improved performance on challenging paraphrase detection tasks.

What Problem Does This Paper Address?

Current paraphrase detection models often rely heavily on lexical and syntactic similarities between sentences. This leads to a "plateauing" effect where models perform well on existing benchmarks but struggle with paraphrases that deviate significantly in wording. The authors argue that true paraphrase detection should focus on inferential properties – whether one sentence can mutually entail the other – rather than just word overlap. Existing datasets, while useful, don't sufficiently challenge models in this regard, creating a need for datasets that specifically target semantically equivalent but structurally different sentence pairs.

Methods

The core of the paper is the Adversarial Paraphrasing Task (APT). This task is designed to generate paraphrases that are:

  1. Mutually Implicative (MI): The sentences must textually entail each other, meaning they have equivalent inferential properties and thus convey the same meaning.
  2. Lexically and Syntactically Disparate: The sentences should have minimal word overlap and structural similarity. This dissimilarity is measured using metrics like BLEURT.

To create datasets for APT, the authors employed two main strategies:

  • Human-driven Generation (APH): They conducted an Amazon Mechanical Turk study where participants were instructed to write sentences that meant the same thing but were structurally different. A reward system incentivized participants to achieve this dissimilarity, with a penalty for high BLEURT scores.
  • Machine-driven Generation (APT5): They fine-tuned a T5base language model using a custom loss function designed to maximize the APT reward. This model was trained on existing paraphrase datasets (TwitterPPDB) and then used to generate new paraphrases that aimed to be APT-passing.

Datasets

The paper introduces and utilizes the following datasets:

  • MSRP (Microsoft Research Paraphrase Corpus): A standard paraphrase dataset used as a source for generating adversarial examples.
  • TwitterPPDB (Paraphrase Database): Another source dataset, derived from Twitter, used for training the T5 model.
  • APH (Adversarial Paraphrase by Humans): A newly created dataset of human-generated paraphrases designed to be semantically equivalent but lexically and syntactically dissimilar.
  • APT5 (T5base-generated APT dataset): A dataset generated by a fine-tuned T5base model, aiming to produce APT-passing paraphrases.

Key Insights and Ideas

  • Meaning vs. Surface Similarity: The paper strongly advocates for moving beyond surface-level lexical and syntactic similarity to assess true paraphrase detection. The APT task directly targets this by forcing models to distinguish meaning from wording.
  • Adversarial Datasets as a Solution: Adversarial data generation is presented as a powerful method to overcome the "plateauing" of performance on existing benchmarks and to create more robust NLP models.
  • Human vs. Machine Performance: While machine-generated datasets (like APT5) can improve model performance, human-generated datasets (like APH) tend to yield even better results, suggesting that human nuance in understanding and generating diverse paraphrases is still superior.
  • BLEURT as a Proxy for Dissimilarity: BLEURT, a metric for evaluating text generation, is repurposed here as a key indicator of lexical and syntactic dissimilarity, crucial for the APT task.

Key Results

  • Poor Performance on Adversarial Data: A state-of-the-art RoBERTa model trained on standard datasets (like TwitterPPDB) performed poorly on the newly created adversarial datasets (APH and APT5), achieving barely better than random accuracy. This highlights the limitations of current models.
  • Significant Improvement with Adversarial Training: When RoBERTa was further trained on the adversarial datasets (APH and APT5), its performance (measured by MCC scores) improved significantly, by up to 0.307.
  • Human Data Outperforms Machine Data: Training with the human-generated APH dataset provided a larger MCC improvement (0.222) compared to training with the machine-generated APT5 dataset (0.188), even though APH was a smaller portion of the training data.
  • Combined Approach is Best: Combining all three adversarial datasets (APH, APT5, and APT) with the baseline TwitterPPDB dataset yielded the highest MCC score (0.525), indicating that both human and machine-generated adversarial data complement each other.

Limitations and Future Work

  • Dataset Imbalance: The paper acknowledges that the generated datasets (especially APH) are imbalanced in terms of MI and non-MI pairs, which is why Matthew's Correlation Coefficient (MCC) is used as a more appropriate evaluation metric than accuracy.
  • Future APT Generation: The authors propose that APT itself can be made more challenging over time. As paraphrase detection models improve, the criteria for APT (e.g., reducing the BLEURT threshold) can be tightened, creating an ongoing adversarial loop for dataset generation.
  • Broader Applications: The authors suggest that improving paraphrase detection has broad applications in areas like healthcare (simplifying medical language), writing (adapting text style), and education (making complex topics more accessible).
  • Exploring Other Metrics: While BLEURT is used, future work could explore other state-of-the-art Semantic Textual Similarity (STS) metrics to measure dissimilarity.

Summary updated

BibTeX
@inproceedings{nighojkar2021improving,
  author = {Animesh Nighojkar and John Licato},
  title = {{Improving Paraphrase Detection with the Adversarial Paraphrasing Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure