Abstract
Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, Kartik Talamadupula. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
Natural Language Inference (NLI)—the task of determining whether a "premise" entails or contradicts a "hypothesis"—is considered a foundational component of natural language understanding. While modern NLI models achieve human-level performance on standard benchmarks, they often fail when applied to real-world downstream tasks such as Multiple-Choice Reading Comprehension (MCRC) or Checking Factual Correctness of Summaries (CFCS). This research identifies a critical bottleneck: the distribution shift in premise length between training data and target applications.
The Premise Length Bottleneck
Traditional NLI datasets, such as SNLI and MNLI, rely on extremely short premises, typically averaging between 14 and 22 words. In contrast, practical applications like reading comprehension or document summarization involve premises (passages or articles) that are significantly longer, often exceeding 500 words.
Standard "divide and conquer" strategies—which split long passages into individual sentences for independent verification—fail to capture the global context and complex reasoning required for high-level understanding. The authors propose the "Long Premise Conjecture," suggesting that NLI models struggle with downstream tasks primarily because they are not trained to reason over extended contexts.
Methodology: Recasting for Long-Premise NLI
To address this gap without the prohibitive cost of manual annotation, the researchers developed a pipeline to automatically "recast" existing MCRC datasets into a long-premise NLI format. The core of this approach is a Hybrid Conversion Strategy designed to transform Question-Answer pairs into declarative hypotheses:
- Rule-based Conversion: Employs dependency parsing to restructure Q+A pairs into statements.
- Neural Generation: Uses a BART model fine-tuned on sentence compression and QA-to-statement datasets to produce natural-sounding hypotheses.
- Hybrid Heuristic: To mitigate neural "hallucinations," the system uses a length-based filter. If the neural output is between 0.8x and 1.2x the length of the original Q+A pair, it is retained; otherwise, the system defaults to the rule-based version.
The researchers then trained a Transferable NLI Model using RoBERTa on these converted datasets, specifically leveraging the RACE dataset due to its high volume (nearly 88,000 instances) and substantial average premise length (~271 words).
Performance Evaluation
The results demonstrate that training on long-premise data significantly enhances a model's ability to generalize to complex tasks. The model trained on converted RACE data ($RACE_{converted}$) consistently outperformed models trained on standard short-premise NLI benchmarks.
On the DREAM reading comprehension dataset, $RACE_{converted}$ achieved 83.58% accuracy, a substantial improvement over MNLI's 67.76%. The benefits extended to summarization tasks as well; on the FactCC benchmark, the $RACE_{converted}$ model reached 86.55% balanced accuracy, far exceeding the 51.92% achieved by the MNLI-trained model and even outperforming the specialized FactCC model (74.15%).
Conclusion
This work demonstrates that the limitations of current NLI models in downstream applications are largely a product of the data they are trained on, rather than inherent architectural flaws. By recasting existing resource-rich datasets like RACE into long-premise NLI formats, we can bridge the gap between sentence-level inference and document-level reasoning. This approach provides a scalable path forward for improving the factual consistency of summarization and the accuracy of reading comprehension systems.
Summary updated