Assessing the Role of Discourse References in Entailment Inference

Shachar Mirkin, Ido Dagan, Sebastian PadóACL 2010

Abstract

This paper investigates the necessity of discourse references—such as coreference and bridging—in Textual Entailment (TE) tasks. Through a manual analysis of the RTE-5 Search task, the authors demonstrate that discourse resolution is mandatory for nearly half of all correct entailment inferences. They propose a framework of tree transformations—Substitution, Merge, and Insertion—to integrate document-level context into logic-based inference engines, arguing that discourse resolution can often serve as a more accessible alternative to extensive world knowledge bases.

Introduction

Textual Entailment (TE) is the foundational NLP task of determining whether a "Hypothesis" (H) can be logically inferred from a given "Text" (T). Traditionally, TE systems have treated sentences as isolated units, a strategy that succeeds when the necessary information is self-contained. However, in real-world applications like the RTE-5 Search task, the "Text" is often a full document where the evidence for an entailment is scattered across multiple sentences.

The authors identify a critical bottleneck in current TE research: systems either ignore discourse context entirely or rely on simplistic "off-the-shelf" coreference resolvers. These tools typically only handle basic noun substitutions (e.g., replacing "he" with "Oswald"). This paper argues for a more systematic understanding of complex discourse phenomena—including bridging references and event coreference—and how they directly impact the inference process.

A Framework for Discourse Integration

To bridge the gap between document-level context and sentence-level logic, the authors propose three formal tree-transformation operations. These operations allow an inference engine to modify the dependency tree of a sentence based on information found elsewhere in the document:

  1. Substitution: The most common approach, where a referring term (such as a pronoun) is replaced by its explicit antecedent.
  2. Merge: This operation is used when information about an entity is distributed across different mentions.
  3. Dependent-Merge combines modifiers from different sentences (e.g., merging "submarine" from one sentence with "AS-28" from another to create the compound "AS-28 submarine").
  4. Head-Merge links a head word to a missing dependent found in the context.
  5. Insertion: Specifically designed for bridging references, this operation creates a new logical path to represent implied relationships, such as location or causality, that are not explicitly stated in the target sentence.

Experimental Methodology

The study utilizes the RTE-5 Search task dataset, which consists of 120 sentence-hypothesis pairs derived from 53 documents. Unlike earlier RTE datasets that provided self-contained snippets, the Search task requires systems to locate entailing sentences within a full corpus, making discourse context unavoidable.

Examples of discourse phenomena in Textual Entailment pairs from the RTE-5 Search task.

To establish an upper bound for performance, the authors conducted a manual, "oracle-style" analysis. Instead of using automated tools, they manually annotated discourse relations to observe how a perfect resolver would assist a TE system. This allowed them to categorize the specific types of references (coreference vs. bridging) and the transformations required to resolve them.

Key Findings and Analysis

The analysis revealed that discourse is not merely a "nice-to-have" feature; it is mandatory. In 44% of the cases studied, resolving discourse references was strictly required to reach the correct entailment decision. Furthermore, the study found that 72% of all analyzed pairs contained relevant discourse references.

One of the most significant insights is the distribution of transformation types. While most existing systems focus solely on substitution, the authors found that nearly half (46%) of the required operations were actually Merge or Insertion. This highlights a major gap in current automated TE pipelines.

Distribution of tree transformation operations across coreference and bridging relations.

Additionally, the authors noted that discourse resolution often acts as a surrogate for expensive world knowledge. For instance, resolving the coreference between "President Kennedy" and a subsequent mention of "Kennedy" is computationally simpler than maintaining a knowledge base that explicitly links every mention of the name to the specific historical figure. The study also highlighted the importance of verbs; approximately 25% of the "focus terms" were events, suggesting that future research must move beyond simple noun coreference to include event-resolution tools.

Conclusion

The primary challenge remains the "Resolution Gap." While this paper provides a roadmap for utilizing discourse information via tree transformations, current automated tools for bridging and event coreference remain significantly weaker than the human "Oracle" used in this study. The authors conclude that discourse resolution should not be treated as a disconnected preprocessing step. Instead, it must be tightly integrated into the inference engine, leading to "inference-aware" resolvers that prioritize the specific structural links—like Merge and Insertion—identified in this research.

Summary updated

BibTeX
@inproceedings{mirkin2010assessing,
  author = {Shachar Mirkin and Ido Dagan and Sebastian Padó},
  title = {{Assessing the Role of Discourse References in Entailment Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
}
Figure