Paraphrase Types for Generation and Detection

Jan Philip Wahle, Bela Gipp, Terry RuasEMNLP 2023 · DOI

Abstract

Paraphrasing has traditionally been treated as a binary problem of semantic similarity. This paper challenges that paradigm by introducing two new tasks: Paraphrase Type Generation (PTG) and Paraphrase Type Detection (PTD). By moving beyond simple "same meaning" labels toward understanding specific linguistic transformations—such as syntax shifts or lexical swaps—the authors demonstrate that "type-aware" models not only provide better interpretability but also achieve superior performance on standard paraphrase benchmarks.

Beyond Binary Similarity

In Natural Language Processing (NLP), paraphrase detection is often reduced to a simple "yes/no" question: are these two sentences semantically equivalent? Similarly, generation tasks typically prioritize high similarity scores (like BLEU or ROUGE) without controlling for how the sentence is transformed.

The authors argue that this shallow approach ignores critical linguistic variables—morphology, syntax, and lexicon. A model that cannot distinguish between a "diathesis alternation" (e.g., active to passive voice) and a "lexicon change" (synonym swap) lacks true linguistic depth. This limitation makes systems like plagiarism detectors less transparent and writing assistants less controllable.

Figure 1: Comparison between simple semantic similarity and fine-grained paraphrase types like Diathesis Alternation and Lexicon Change.

Formalizing Paraphrase Types

To bridge this gap, the researchers formalize two distinct tasks designed to force models to recognize and manipulate specific linguistic variables:

  1. Paraphrase Type Generation (PTG): Given a source sentence and a target transformation type (e.g., "change the syntax"), the model must generate a corresponding paraphrase.
  2. Paraphrase Type Detection (PTD): Given a pair of sentences, the model must identify the specific linguistic transformations that occurred and pinpoint where they happened.
Figure 2: Workflows for Paraphrase Type Generation (left) and Paraphrase Type Detection (right).

Methodology and Architectures

The study evaluated several architectures across these tasks:

  • Encoder-Decoders: BART and PEGASUS were fine-tuned for the generation task (PTG).
  • Autoencoders: BERT, RoBERTa, ERNIE, and DeBERTa were adapted with linear layers for token-level classification in the detection task (PTD).
  • Large Language Models (LLMs): LLaMA (7B to 65B parameters) and ChatGPT-3.5 were tested using in-context learning to assess how general-purpose models handle fine-grained linguistic instructions.

Datasets and Benchmarks

The primary training resource was the Extended Paraphrase Typology Corpus (ETPC), which categorizes paraphrases into 26 fine-grained types across six high-level categories (e.g., Morphology-based, Syntax-based). To test the transferability of "type-awareness," the models were also evaluated on broader benchmarks: QQP (Quora Question Pairs), TURL (Twitter News), and PAWS (Paraphrase Adversaries from Word Scrambling).

Key Results

  • Detection Performance: ChatGPT-3.5 led the detection task with 90.4% accuracy in binary scenarios, though this fell to 76.8% when identifying specific types. Among specialized models, DeBERTa proved the most robust.
  • Generation Quality: BART outperformed PEGASUS, particularly in ROUGE-L (54.2), indicating a stronger ability to handle complex contexts.
  • The "Type-Awareness" Boost: Training models to recognize specific types significantly improved their general performance. For instance, BART’s ROUGE-L score on the QQP dataset rose from 41.8 to 44.2 when it was made type-aware.
  • Scaling to Human Levels: Scaling LLaMA to 65B parameters allowed it to surpass reported human performance levels on QQP, TURL, and PAWS when provided with explicit paraphrase type instructions.

Discussion and Future Work

While the results are promising, the authors identify several areas for growth. The ETPC dataset is relatively small (~5,800 sentences), suggesting a need for LLM-driven data augmentation. Furthermore, standard metrics like BLEU and ROUGE are insufficient for measuring whether a model successfully executed a specific requested change; new, type-specific metrics are required.

Ultimately, this research provides a roadmap for more transparent AI. Instead of a black-box similarity score, future systems could explain their reasoning: "This sentence was flagged because the lexicon was changed and the voice was shifted from active to passive."

Summary updated

BibTeX
@inproceedings{wahle2023paraphrase,
  author = {Jan Philip Wahle and Bela Gipp and Terry Ruas},
  title = {{Paraphrase Types for Generation and Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.746},
}
Figure