Recovering Lexically and Semantically Reused Texts

Ansel MacLaughlin, Shaobin Xu, David A. SmithACL 2021

TL;DR

This paper tackles the challenge of detecting localized text reuse (plagiarism, paraphrasing, citation) within documents, going beyond simple document-level similarity. They conduct a large-scale evaluation of various neural and bag-of-words models, introducing a new citation localization dataset and offering insights into model performance, context utilization, and the trade-offs between efficiency and accuracy.

What Problem Does This Paper Address?

When people write, they often reuse text from existing sources. This can range from direct quotes and paraphrases to more subtle semantic reuse, like summarizing a paper in an academic citation. While document-level similarity models can tell us if two texts are generally alike, they struggle to pinpoint where and how specific passages are reused within larger, otherwise unrelated documents. This "local text reuse detection" (LTRD) is crucial for understanding information flow, detecting plagiarism, and analyzing how ideas spread. The paper aims to systematically evaluate different computational approaches to this problem, especially focusing on the effectiveness of modern neural models.

Methods

The authors compare four classes of models for LTRD:

  • Lexical Overlap Models: These are traditional methods like TF-IDF cosine similarity and ROUGE scores. They are computationally efficient and serve as strong baselines.
  • Pretrained Sentence-BERT Encoders (SBERT): These models leverage BERT's power for semantic understanding but are trained on general tasks like paraphrase detection (PD) or semantic textual similarity (STS), not specifically LTRD. The paper evaluates how well these "off-the-shelf" models generalize.
  • Hierarchical Neural Models (HNM): These models, including BERT-HAN and GRU-HAN, use hierarchical attention mechanisms to capture document-level context and cross-document alignment. They often use frozen embeddings for efficiency.
  • Fine-tuned BERT-based Models: These are state-of-the-art models (like RoBERTa and Longformer) that are fine-tuned directly on LTRD tasks. They require more computational resources as they involve separate passes for each sentence pair.

The paper also explores domain-adaptive pretraining (DAPT) to see if adapting models to specific domains improves performance.

Datasets

The study uses four datasets:

  • PAN and S2ORC: These are existing benchmark LTRD datasets for plagiarism detection and citation localization, respectively.
  • Pr2News: A dataset of text reuse in news articles, labeled with a mix of expert and heuristic annotations. It's notable for its weakly-supervised training data.
  • ARC-Sim: A new, publicly available citation localization dataset created from the ACL Anthology. It's designed to have "hard negative" examples, making it more challenging than S2ORC.

Key Insights and Ideas

  • Fine-tuned BERT reigns supreme: Models fine-tuned specifically for LTRD consistently outperform other approaches, especially on tasks involving non-literal reuse (paraphrasing, summarization).
  • Context matters, but not always: While models that incorporate document-level context (like Longformer) can improve performance, the benefit isn't universal. Sometimes, simpler models or those focusing on sentence pairs perform better, especially on document-level metrics.
  • Off-the-shelf models struggle: General-purpose semantic models like SBERT, trained on tasks like STS or PD, often perform worse than simpler lexical baselines when applied directly to LTRD without fine-tuning.
  • Hard negatives are challenging: The new ARC-Sim dataset, with its carefully constructed negative examples, proves more difficult for models, highlighting the need for robust LTRD methods that can distinguish subtle reuse from unrelated text.
  • Efficiency vs. Accuracy Trade-off: Lexical overlap methods are fast and provide good baselines, while fine-tuned BERT models offer the highest accuracy but at a significant computational cost.

Key Results

  • Fine-tuned BERT models achieved the best performance across most datasets and metrics, particularly on S2ORC and ARC-Sim, outperforming the next best models by significant margins (e.g., 6.3% for D2D and 15.5% for S2D on S2ORC).
  • Lexical overlap models (TF-IDF, ROUGE) provided surprisingly strong baselines, often outperforming off-the-shelf SBERT models and hierarchical models on some tasks.
  • On the PAN dataset, which features many direct copies and topically unrelated negative pairs, models achieved very high scores. Conversely, ARC-Sim, with its harder negatives, showed lower performance across the board.
  • Domain-adaptive pretraining (DAPT) showed mixed results, outperforming standard models on S2ORC but not on ARC-Sim or Pr2News, suggesting its effectiveness is highly dependent on the domain and task.

Limitations and Future Work

  • Weak Supervision: The Pr2News dataset's reliance on heuristic labels for training presents a challenge, and the authors suggest exploring larger weakly-supervised datasets and combining multiple heuristics for better supervision signals.
  • Computational Cost: Fine-tuned BERT models, while achieving the best performance, are computationally expensive, making them potentially infeasible for very large-scale applications.
  • Dataset Limitations: The authors note that the Pr2News dataset is smaller due to human annotation, and that S2ORC's assumption that citing sentences often paraphrase abstracts might not always hold. They also suggest improvements for ARC-Sim, such as manual annotation of reused sentences without explicit citation marks and better OCR.
  • Future Directions: The paper suggests combining efficient lexical overlap methods for initial filtering with more expensive fine-tuned BERT models for reranking. They also highlight the potential of models that incorporate document-level context (like Longformer) and domain-adapted models for further performance gains.

Summary updated

BibTeX
@inproceedings{maclaughlin2021recovering,
  author = {Ansel MacLaughlin and Shaobin Xu and David A. Smith},
  title = {{Recovering Lexically and Semantically Reused Texts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure