R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference

Hao Wang, Yixin Cao, Yangguang Li, Zhen Huang, Kun Wang, Jing Shao

Introduction

Natural Language Inference (NLI) is a fundamental task in natural language processing that involves determining the logical relationship between a premise and a hypothesis. While sentence-level NLI has seen significant progress, Document-level NLI (DOCNLI) presents unique challenges. Standard Transformer-based models often struggle with DOCNLI due to fixed token limits (typically 512 tokens) and the high "noise floor" inherent in long documents, where critical evidence may be buried within irrelevant context. Furthermore, these models often operate as "black boxes," providing little explanation for their predictions.

To address these issues, the R²F framework adopts a "divide and conquer" strategy. By decomposing document-level tasks into sentence-level components, R²F effectively handles long-range dependencies, filters out noise, and provides human-readable explanations for its logical conclusions.

The R²F Framework

The R²F framework consists of three primary stages: Retrieval, Reading, and Fusion. This pipeline transforms the document-level inference problem into a series of manageable sub-tasks.

The R²F framework overview: Retrieval, Reading, and Fusion stages.
  1. Retrieval: For each sentence in the hypothesis document, the system retrieves the top $K$ most relevant sentences from the premise document using methods such as BM25 or SimCSE. This stage acts as a denoising filter, ensuring the model focuses only on relevant evidence.
  2. Reading: A RoBERTa-based reader processes each hypothesis sentence alongside its retrieved evidence to calculate a credibility score. This score represents the likelihood that the specific hypothesis sentence is supported by the premise.
  3. Fusion: To determine the relationship for the entire document, the framework employs Credibility Score Minimum Pooling. This logic follows the "weakest link" principle: if any single sentence in the hypothesis is contradicted or unsupported, the entire document relationship is classified as "not entailment."

Datasets and Evaluation

The framework was evaluated primarily on the DOCNLI dataset, which contains over one million document pairs. To further assess model interpretability, the authors introduced a Complementary Annotation subset. This subset includes 100 long-form samples (over 800 words) with manual annotations identifying the specific premise sentences that serve as evidence for each hypothesis sentence. This benchmark allows for a more granular evaluation of how well the model identifies supporting facts.

Key Results

The R²F framework achieved state-of-the-art performance on the DOCNLI dataset. Using a RoBERTa-large backbone, it reached a Macro F1 of 78.48, significantly outperforming the standard concatenation baseline of 75.66.

Performance comparison of different fusion strategies on the DOCNLI dataset.

Key findings from the experimental results include:

  • Scalability to Long Documents: The performance advantage of R²F becomes more pronounced as document length increases. On documents exceeding 1,000 words, standard models often see performance drops of over 10% due to truncation, while R²F remains robust.
  • Effective Fusion Logic: Comparative analysis of fusion strategies (as seen in Figure 6) demonstrates that the Credibility Score approach provides superior results compared to using raw inference vectors or Gaussian kernels, particularly in Macro F1 metrics.
  • Interpretability: By generating scores for individual sentences, the model can pinpoint exactly which part of a document led to a specific verdict, such as identifying a factual mismatch or a missing piece of evidence.

Conclusion and Future Work

R²F demonstrates that a modular pipeline can overcome the structural limitations of standard Transformers in document-level tasks. While the framework is highly effective, the authors note that its performance is dependent on the precision of the retrieval stage. Future research may explore end-to-end training techniques, such as reinforcement learning, to allow the Reader and Retriever to optimize jointly for better evidence discovery.

Summary updated

Abstract

Document-level natural language inference (DOCNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, but fail to address the issues raised by longer documents. In this paper, we establish a general solution, named Retrieval, Reading and Fusion (R2F) framework, and a new setting, by analyzing the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference. The basic idea of the framework is to simplify document-level task into a set of sentence-level tasks, and improve both performance and interpretability with the power of evidence. For each hypothesis sentence, the framework retrieves evidence sentences from the premise, and reads to estimate its credibility. Then the sentence-level results are fused to judge the relationship between the documents. For the setting, we contribute complementary evidence and entailment label annotation on hypothesis sentences, for interpretability study. Our experimental results show that R2F framework can obtain state-of-the-art performance and is robust for diverse evidence retrieval methods. Moreover, it can give more interpretable prediction results. Our model and code are released at https://github.com/phoenixsecularbird/R2F.
BibTeX
@inproceedings{wang2022r,
  author = {Hao Wang and Yixin Cao and Yangguang Li and Zhen Huang and Kun Wang and Jing Shao},
  title = {{R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=VWZUEoJo4VO},
}
Figure