TRoTR: A Framework for Evaluating the Re-contextualization of Text Reuse

Francesco Periti, Pierluigi Cassotti, Stefano Montanelli, Nina Tahmasebi, Dominik SchlechtwegEMNLP 2024 · DOI

Introduction: Beyond Simple Detection

In Natural Language Processing (NLP), Text Reuse Detection (TRD) is a mature field used for everything from plagiarism checking to tracking the spread of news. However, traditional TRD often operates under a significant assumption: that if a text is reused, it must be topically related to its source.

The TRoTR (Topic Relatedness of Text Reuse) framework challenges this assumption. A single sentence—such as a biblical verse or a political slogan—can be "recontextualized" to support vastly different, even contradictory, ideas. For instance, a quote about "laying down one's life" can be used to celebrate personal sacrifice in a social context or to justify military action in a political one. While the words remain identical, the topics diverge.

Examples of recontextualized text reuse showing how identical quotes can appear in unrelated topics.

The TRoTR Framework

To address this gap, the researchers proposed two distinct tasks designed to measure how well models understand the context surrounding reused text:

  1. Text Reuse in-Context (TRiC): A pair-level task where a model determines if two instances of the same reused text share the same topic (binary classification) and ranks them by their degree of relatedness.
  2. Topic variation Ranking across Corpus (TRaC): A corpus-level task where the goal is to rank different reused passages based on the degree of topic variation they exhibit across an entire dataset.
The TRoTR framework: defining the TRiC and TRaC tasks alongside the human annotation process using the DURel scale.

The TRoTR Benchmark Dataset

The authors introduced a new human-annotated benchmark focused on biblical text reuse on Twitter. Biblical text is particularly suited for this study due to its high "context variety."

  • Data Source: 42 target biblical passages, each associated with 30 tweets, totaling 1,260 tweets.
  • Annotation: Human annotators rated pairs of tweets on the 4-point DURel scale, ranging from "Identical Topic" (4) to "Unrelated" (1).
  • Quality: The final dataset contains 3,821 annotated context pairs with a high inter-annotator agreement (Spearman's $\rho$ of .811).

Experimental Evaluation

The researchers evaluated 36 SBERT (Sentence-BERT) models across four experimental settings to see how they handle recontextualization:

  • Base: Off-the-shelf pre-trained models.
  • +MASK: Replacing the reused text with a placeholder ("-") to force the model to focus exclusively on the surrounding context.
  • +FT: Fine-tuning models specifically on the TRiC task.
  • +FT+MASK: Combining fine-tuning with the masking strategy.

Key Insights: Overcoming the "Shared Substring" Bias

The study revealed that standard pre-trained models suffer from a massive shared substring bias. Because the reused quote is identical in both samples, models tend to rate them as highly related, regardless of the surrounding context. They essentially become "blinded" by the surface-level word matching.

Interestingly, masking the reused text proved to be a powerful strategy. By hiding the identical quote, models were forced to "read" the context, significantly improving their ability to identify topic shifts. Additionally, while Cross-Encoders are often preferred for similarity tasks, they were more prone to overfitting on the shared text in this scenario; Bi-Encoders proved more robust.

Results and Future Directions

Fine-tuning was essential for success. Off-the-shelf models showed high precision but very low recall for "different topics"—they almost never correctly identified when a quote was being used in a new, unrelated context. Fine-tuning on the TRiC task boosted F1-scores from ~0.61 to ~0.73. In the TRaC corpus ranking task, the +MASK setting improved Spearman correlations from the .65–.73 range to .76–.84.

Future work aims to expand the TRoTR benchmark beyond English and biblical texts to include political slogans and proverbs. The researchers also plan to explore "diachronic evolution"—how the topical use of famous literary quotes has changed over centuries and across different cultures.

Summary updated

BibTeX
@inproceedings{periti2024trotr,
  author = {Francesco Periti and Pierluigi Cassotti and Stefano Montanelli and Nina Tahmasebi and Dominik Schlechtweg},
  title = {{TRoTR: A Framework for Evaluating the Re-contextualization of Text Reuse}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.774},
}
Figure