Task-Oriented Paraphrase Analytics

Marcel Gohsen, Matthias Hagen, Martin Potthast, Benno Stein

TL;DR

Paraphrasing is often treated as a monolithic task in NLP, but it actually encompasses a diverse set of 25 distinct sub-tasks ranging from sentence compression to author obfuscation. This paper introduces a formal taxonomy for these tasks and reveals that popular "general-purpose" paraphrase datasets are heavily biased toward specific sub-tasks, meaning models trained on them may not generalize as well as researchers assume.

What Problem Does This Paper Address?

In the world of Natural Language Processing (NLP), the definition of a "paraphrase" is notoriously fuzzy. Definitions range from "conveying the same meaning with different words" to "talking about the same situation in a different way." Because the term is so broad, researchers often use any available paraphrase corpus to train or evaluate their models, assuming these datasets represent a homogeneous phenomenon.

The authors argue that this is a mistake. A paraphrase generated for text simplification (making a sentence easier to read) has very different characteristics than one generated for author obfuscation (hiding a writer's identity) or image recaptioning. By treating all paraphrases the same, we risk creating models that are "jacks of all trades but masters of none," and we produce evaluation results that are incomparable across different datasets.

Methods

The researchers tackled this problem through a three-step approach:

  1. Taxonomy Construction: Through an extensive literature review, they identified 25 distinct paraphrasing sub-tasks. They organized these into a taxonomy based on whether the task requires semantic equivalence (the meaning stays exactly the same) or merely semantic similarity (the meaning is mostly preserved but might shift slightly).
  2. Human Validation: To see if these categories actually mean anything to people, they had an expert annotator classify 500 paraphrase pairs across five major sub-tasks.
  3. Automatic Classification: They built a Random Forest classifier to automatically identify which sub-task a paraphrase pair belongs to. They used a mix of standard features:
  4. Compression Ratio: The length difference between the two texts.
  5. Surface Similarity: ROUGE and BLEU scores.
  6. Semantic Similarity: Cosine similarity of Sentence-BERT embeddings.
  7. Syntactic Structure: Vectorized relative frequencies of POS tag n-grams.

Finally, they used this classifier to perform "Paraphrase Corpora Analytics," measuring the distribution of these tasks across four widely used general-purpose datasets: MSRPC, PAWS, TaPaCo, and Wikipedia-IPC.

Datasets

The authors utilized a wide array of datasets to train their task-specific classifier:

  • Simplification: TurkCorpus and WikiLarge.
  • Compression: Google’s compression dataset and Microsoft’s abstractive compression dataset.
  • Style Transfer: ParaDetox (toxic to non-toxic) and a Bible style transfer dataset.
  • Image Recaptioning: MSCOCO and VizWiz.
  • Textual Entailment: SciTail and HELP.

They then analyzed the "task bias" of four major general-purpose benchmarks: MSRPC, PAWS, TaPaCo, and Wikipedia-IPC.

Key Insights and Ideas

  • The Equivalence vs. Similarity Divide: The authors provide a clear roadmap for the field by splitting tasks into two camps. Figure 1 illustrates tasks requiring strict equivalence, like Acrostification or Copy Editing.
Figure 1: Taxonomy of paraphrase generation tasks which require generated paraphrases to be semanti- cally equivalent to the original text.
  • Datasets are Highly Biased: The most striking "aha moment" is that "general-purpose" datasets aren't general at all. For example, a researcher using the TaPaCo dataset is actually training their model on 76.8% style transfer instances. Meanwhile, PAWS is dominated by simplification (62.7%).
  • The "Unknown" Factor: In human trials, many "unknown" classifications came from corpus artifacts—misaligned sentences where the meaning didn't match at all. This highlights the noise present in even standard benchmarks.
  • Task Confusion: Both humans and machines frequently confuse simplification, compression, and entailment. This makes sense: all three often involve shortening a sentence or removing specific details, showing that these sub-tasks share significant underlying signals.

Key Results

The study proved that humans can distinguish between these tasks with high accuracy (100% for captions, 94% for style transfer). The automatic classifier also performed well, achieving a micro-averaged F1 score of 0.82.

The most important quantitative finding is the distribution of tasks across standard corpora (Table 1 in the paper):

  • Wikipedia-IPC: Heavily skewed toward Sentence Compression (62.0%).
  • TaPaCo: Overwhelmingly Style Transfer (76.8%).
  • MSRPC: The most diverse of the bunch, but still shows a bias toward compression and simplification.
Figure 4: Confusion matrix of the automatically predicted paraphrasing tasks and the actual tasks of paraphrases from task-specific corpora in the sampled test-set.

As seen in the confusion matrix above (Figure 4 in the paper), the classifier is particularly robust at identifying image recaptioning and style transfer, though it occasionally struggles with the nuances between compression and simplification.

Limitations and Future Work

The authors acknowledge that their classifier is currently single-label, whereas many paraphrases might actually fit multiple categories (e.g., a sentence could be both simplified and compressed). They suggest that future work should move toward multi-label classification, though they note that obtaining high-quality training data for this is significantly harder.

Additionally, the classifier's generalizability is still being tested. The fact that it identified a low number of "image captions" in a dataset actually composed of captions (Wikipedia-IPC) suggests that the model might be picking up on dataset-specific quirks rather than pure task-level features. The authors call for more manual annotation of general-purpose datasets to verify these automated findings.

Summary updated

Abstract

Since paraphrasing is an ill-defined task, the term "paraphrasing" covers text transformation tasks with different characteristics. Consequently, existing paraphrasing studies have applied quite different (explicit and implicit) criteria as to when a pair of texts is to be considered a paraphrase, all of which amount to postulating a certain level of semantic or lexical similarity. In this paper, we conduct a literature review and propose a taxonomy to organize the 25~identified paraphrasing (sub-)tasks. Using classifiers trained to identify the tasks that a given paraphrasing instance fits, we find that the distributions of task-specific instances in the known paraphrase corpora vary substantially. This means that the use of these corpora, without the respective paraphrase conditions being clearly defined (which is the normal case), must lead to incomparable and misleading results.
BibTeX
@inproceedings{gohsen2024task,
  author = {Marcel Gohsen and Matthias Hagen and Martin Potthast and Benno Stein},
  title = {{Task-Oriented Paraphrase Analytics}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure