Abstract
Traditional Natural Language Processing (NLP) often treats paraphrasing as a context-independent semantic relationship. However, in human dialog, paraphrasing is a dynamic act where meaning is negotiated across turns. This paper introduces ContextDeP, a novel dataset and framework specifically designed to identify "context-dependent paraphrases" (CDPs) in news interviews. We operationalize the definition of CDPs, provide a high-density annotated corpus, and benchmark both Large Language Models (LLMs) and token classification models. Our findings reveal a significant trade-off: while LLMs excel at high-level detection, they frequently struggle with precise extraction and hallucination.
Introduction
In standard NLP tasks, paraphrasing is typically viewed as a static equivalence between two sentences. For instance, "The movie was great" and "The film was excellent" are considered paraphrases regardless of the surrounding conversation. Yet, real-world dialog is rarely so isolated. In a news interview, a host might paraphrase a guest's statement to summarize, clarify, or shift perspective. In these cases, the equivalence is context-dependent. A phrase like "I am happy" and "You are feeling good" are only equivalent because of the specific roles of the speakers at that moment.
Existing datasets often fail to capture these nuances because they focus on full-sentence pairs in isolation, ignoring the "across-turns" nature of dialog where a paraphrase might be a small snippet buried within a long response. This research addresses this gap by defining, annotating, and detecting these elusive, context-heavy paraphrases.
Defining Context-Dependent Paraphrases
The authors define a context-dependent paraphrase (CDP) as two text excerpts that are approximately equivalent within a specific situational context, even if they lack equivalence in a vacuum. This definition encompasses both clear semantic matches and more approximate equivalences that arise from the flow of conversation.
To capture these, the researchers developed a multi-stage approach. Recognizing the difficulty of the task, they implemented an interactive, 15-minute "example-centric" training program for crowd-workers. Annotators were required to pass comprehension checks and demonstrate the ability to identify specific spans of text rather than simply labeling entire turns.
The ContextDeP Dataset
The resulting dataset, ContextDeP, is built from the MediaSum corpus, consisting of NPR and CNN news interviews. It contains 600 unique utterance pairs with a total of 5,581 individual annotations.
A key feature of ContextDeP is its annotation density. While most datasets provide only 1–3 labels per pair, ContextDeP provides up to 21 annotations per item. This high density allows for the study of plausible label variation—the concept that disagreement among human annotators often reflects genuine linguistic ambiguity rather than error.
The data is categorized into three subsets: RANDOM (uniform sampling), BALANCED (50/50 split), and PARA (high-density paraphrase) to ensure robust model testing across different scenarios.
Modeling and Experimental Setup
The study benchmarks two primary modeling paradigms:
- In-Context Learning (ICL): Utilizing Large Language Models like GPT-4 and Llama 2 in a few-shot setup to both classify the presence of a paraphrase and highlight the relevant spans.
- Token Classification: Fine-tuning a DeBERTa-v3-large model to predict, on a word-by-word basis, whether a token belongs to a paraphrase span.
Key Results and Discussion
The experiments revealed a distinct performance trade-off between generative and discriminative models. GPT-4 achieved the highest F1 score for classification (0.81), outperforming the fine-tuned DeBERTa model (0.73). However, GPT-4 struggled significantly with the "where" of the task. It exhibited a high "extraction error" rate (up to 71%), often hallucinating spans by slightly altering words or misquoting the speaker.
In contrast, the fine-tuned DeBERTa model was the most reliable for identifying exact word spans (measured by the Jaccard Index), avoiding the formatting and hallucination issues inherent in generative LLMs.
Furthermore, the study highlights the inherent subjectivity of the task. Inter-annotator agreement (Krippendorff’s alpha) ranged from 0.19 to 0.32. Rather than viewing this as a failure, the authors argue that "ground truth" in dialog is often a distribution of opinions. This suggests that future evaluation metrics should move toward "soft" scoring that rewards models for capturing the breadth of human interpretation.
Conclusion
ContextDeP provides a foundational framework for understanding how meaning is shared and transformed in dialog. By moving beyond context-independent sentence pairs, this work opens new avenues for studying the dynamic negotiation of meaning in complex human interactions. Future work will look to expand this approach to other domains, such as clinical counseling, and explore metrics that better account for the inherent subjectivity of human conversation.
Summary updated