Abstract
Document-level natural language inference (DOCNLI) is a challenging task in natural language processing that involves judging the entailment relationship between a pair of hypothesis and premise documents. Current datasets and baselines largely follow sentence-level settings, failing to address the complexities of longer documents. In this paper, we establish a general solution named the Retrieval, Reading and Fusion (R2F) framework. By simplifying document-level tasks into a set of sentence-level tasks, R2F addresses the main challenges of DOCNLI: interpretability, long-range dependency, and cross-sentence inference. Our experimental results show that the R2F framework obtains state-of-the-art performance, remains robust across diverse evidence retrieval methods, and provides highly interpretable prediction results.
Introduction
Natural Language Inference (NLI)—the task of determining if a "premise" entails a "hypothesis"—has long been a staple of NLP. However, most research has focused on sentence-level pairs. When scaled to entire documents, the complexity increases significantly. Standard NLI models usually concatenate a premise and a hypothesis and feed them into a transformer like BERT or RoBERTa. While effective for sentences, this approach fails for documents due to three primary reasons:
- Long-range Dependency: Documents often exceed the 512-token limit of standard Pre-trained Language Models (PLMs). Truncating text results in the loss of vital information.
- Cross-sentence Inference: A single hypothesis sentence might require evidence scattered across multiple, non-contiguous sentences in the premise.
- Interpretability: If a model determines a 1,000-word document does not entail another, users need to know why. Current "black-box" models cannot easily point to the specific sentence that caused the contradiction.
To tackle these unique challenges, we introduce the R$^2$F (Retrieval, Reading, and Fusion) framework, a "divide and conquer" strategy for DOCNLI.
The R$^2$F Framework
The R$^2$F framework operates in three distinct stages to decompose the document-level task into sentence-level components:
1. Retrieval
For every sentence in the hypothesis document, the system searches the premise document for the top $K$ most relevant "evidence" sentences. The framework is designed to be robust, supporting various retrieval methods including sparse retrieval (BM25, ROUGE-1) and dense retrieval (SimCSE).
2. Reading
The hypothesis sentence and its retrieved evidence are fed into a reader model (such as RoBERTa). Instead of a simple binary label, the model outputs a credibility score ($\hat{y}_i$) between 0 and 1, representing the likelihood that the specific hypothesis sentence is supported by the premise.
3. Fusion
To reach a final document-level verdict, the model fuses the individual sentence scores. The primary method employed is Credibility Score Minimum Pooling. The logic is intuitive: in the context of entailment, a single piece of disinformation or an unsupported claim (the "weakest link") is sufficient to invalidate the entire document-level relationship.
Experimental Setup and Results
We evaluated R$^2$F using the DOCNLI dataset, a large-scale benchmark containing over 1 million document pairs. To further study interpretability, we manually annotated a subset of 100 long samples (containing over 350 hypothesis sentences) to identify specific "evidence groups."
Performance and Scalability
R$^2$F achieved state-of-the-art performance on the DOCNLI dataset. Using RoBERTa-large, it reached a Macro F1 of 78.48, significantly outperforming the standard concatenation approach (75.66). Notably, the performance gap widened as document length increased. On samples exceeding 1,000 words, R$^2$F maintained high accuracy while standard models struggled due to input truncation.
Interpretability and Robustness
Because the model generates a score for every hypothesis sentence, it provides a clear map of its reasoning. In case studies, the model successfully identified specific factual discrepancies (such as incorrect dates) as the sole reason for a "not entailment" label. Furthermore, the framework proved robust to retrieval methods; surprisingly, simple sparse retrieval (ROUGE-1) performed comparably to complex dense retrieval, suggesting that literal word overlap remains a strong signal for evidence in NLI tasks.
Conclusion
The R$^2$F framework provides a scalable and interpretable solution for document-level NLI. By breaking the problem down into retrieval, reading, and fusion, it overcomes the token-limit constraints of standard transformers and offers a logical path toward verifiable AI reasoning. Future work will focus on developing end-to-end versions of the framework to optimize retrieval and reading simultaneously and addressing potential evidence dependency biases.
Summary updated