R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference

Hao Wang, Yixin Cao, Yangguang Li, Zhen Huang, Kun Wang, Jing Shao

Abstract

Document-level natural language inference (DOCNLI) is a challenging task in natural language processing that involves judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, failing to address the complexities of longer documents. In this paper, we establish a general solution named the Retrieval, Reading and Fusion (R2F) framework. By simplifying document-level tasks into a set of sentence-level tasks, R2F addresses the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference. Our experimental results show that the R2F framework obtains state-of-the-art performance, remains robust across diverse evidence retrieval methods, and provides highly interpretable prediction results.

Introduction

Natural Language Inference (NLI)—the task of determining if a "premise" entails a "hypothesis"—has long been a staple of NLP. However, most research has focused on sentence-level pairs. When scaled to entire documents, the complexity increases significantly. Standard NLI models usually concatenate a premise and a hypothesis and feed them into a transformer like BERT or RoBERTa. While effective for sentences, this approach fails for documents due to three primary reasons:

  1. Long-range Dependency: Documents often exceed the 512-token limit of standard Pre-trained Language Models (PLMs). Truncating text results in the loss of vital information.
  2. Cross-sentence Inference: A single hypothesis sentence might require evidence scattered across multiple, non-contiguous sentences in the premise.
  3. Interpretability: If a model determines a 1,000-word document does not entail another, users need to know why. Current "black-box" models cannot easily point to the specific sentence that caused the contradiction.

To tackle these unique challenges, we introduce the R$^2$F (Retrieval, Reading, and Fusion) framework, a "divide and conquer" strategy for DOCNLI.

The R$^2$F Framework

The R$^2$F framework operates in three distinct stages to decompose the document-level task into sentence-level components:

1. Retrieval

For every sentence in the hypothesis document, the system searches the premise document for the top $K$ most relevant "evidence" sentences. The framework is designed to be robust, supporting various retrieval methods including sparse retrieval (BM25, ROUGE-1) and dense retrieval (SimCSE).

2. Reading

The hypothesis sentence and its retrieved evidence are fed into a reader model (such as RoBERTa). Instead of a simple binary label, the model outputs a credibility score ($\hat{y}_i$) between 0 and 1, representing the likelihood that the specific hypothesis sentence is supported by the premise.

3. Fusion

To reach a final document-level verdict, the model fuses the individual sentence scores. The primary method employed is Credibility Score Minimum Pooling. The logic is intuitive: in the context of entailment, a single piece of disinformation or an unsupported claim (the "weakest link") is sufficient to invalidate the entire document-level relationship.

Experimental Setup and Results

We evaluated R$^2$F using the DOCNLI dataset, a large-scale benchmark containing over 1 million document pairs. To further study interpretability, we manually annotated a subset of 100 long samples (containing over 350 hypothesis sentences) to identify specific "evidence groups."

Performance and Scalability

R$^2$F achieved state-of-the-art performance on the DOCNLI dataset. Using RoBERTa-large, it reached a Macro F1 of 78.48, significantly outperforming the standard concatenation approach (75.66). Notably, the performance gap widened as document length increased. On samples exceeding 1,000 words, R$^2$F maintained high accuracy while standard models struggled due to input truncation.

Interpretability and Robustness

Because the model generates a score for every hypothesis sentence, it provides a clear map of its reasoning. In case studies, the model successfully identified specific factual discrepancies (such as incorrect dates) as the sole reason for a "not entailment" label. Furthermore, the framework proved robust to retrieval methods; surprisingly, simple sparse retrieval (ROUGE-1) performed comparably to complex dense retrieval, suggesting that literal word overlap remains a strong signal for evidence in NLI tasks.

Conclusion

The R$^2$F framework provides a scalable and interpretable solution for document-level NLI. By breaking the problem down into retrieval, reading, and fusion, it overcomes the token-limit constraints of standard transformers and offers a logical path toward verifiable AI reasoning. Future work will focus on developing end-to-end versions of the framework to optimize retrieval and reading simultaneously and addressing potential evidence dependency biases.

Summary updated

Abstract

Document-level natural language inference (DOCNLI) is a new challenging task in natural language processing, aiming at judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, but fail to address the issues raised by longer documents. In this paper, we establish a general solution, named Retrieval, Reading and Fusion (R2F) framework, and a new setting, by analyzing the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference. The basic idea of the framework is to simplify document-level task into a set of sentence-level tasks, and improve both performance and interpretability with the power of evidence. For each hypothesis sentence, the framework retrieves evidence sentences from the premise, and reads to estimate its credibility. Then the sentence-level results are fused to judge the relationship between the documents. For the setting, we contribute complementary evidence and entailment label annotation on hypothesis sentences, for interpretability study. Our experimental results show that R2F framework can obtain state-of-the-art performance and is robust for diverse evidence retrieval methods. Moreover, it can give more interpretable prediction results. Our model and code are released at https://github.com/phoenixsecularbird/R2F.
BibTeX
@inproceedings{wang2022r,
  author = {Hao Wang and Yixin Cao and Yangguang Li and Zhen Huang and Kun Wang and Jing Shao},
  title = {{R2F: A General Retrieval, Reading and Fusion Framework for Document-level Natural Language Inference}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=VWZUEoJo4VO},
}
Figure