Abstract
Large Language Models (LLMs) and Question Answering (QA) systems frequently generate "hallucinations"—answers that are fluent but factually incorrect. While Natural Language Inference (NLI) is often employed to verify these answers by checking for entailment (support) from a given context, this research highlights a critical oversight: the value of the contradiction signal. This paper demonstrates that explicitly checking if a context refutes a candidate answer significantly improves performance across 16 datasets. The findings suggest that "not being wrong" is often a more reliable indicator of correctness than "having some support," particularly in high-stakes selective QA scenarios.
Introduction
The deployment of AI in sensitive domains like medicine or law is hindered by the tendency of models to produce unsupported claims. To mitigate this, researchers use Natural Language Inference (NLI) to determine the relationship between a premise (the context) and a hypothesis (the proposed answer). Historically, the focus has been almost exclusively on the "entailment" score—the degree to which the context supports the answer.
However, the authors of "Using contradictions improves question answering systems" argue that the "contradiction" score is a vital, underutilized signal. An answer that lacks explicit support might still be correct, but an answer that is explicitly contradicted by the text is a clear error that must be filtered out. By incorporating this negative signal, QA systems can become both more accurate and significantly safer.
Methodology
The researchers proposed a multi-stage pipeline to incorporate contradiction signals into standard QA workflows:
- Answer Generation: A base QA model (e.g., RoBERTa or DeBERTa) generates candidate answers and provides initial confidence scores (QA).
- QA2D (Question-Answer to Declarative): NLI models require full sentences. The authors used a T5-small model to transform (Question, Answer) pairs into declarative statements. For example, the question "What is the most abundant metal?" paired with the answer "Aluminum" becomes "The most abundant metal is aluminum."
- NLI Scoring: These statements are compared against the context using an NLI model to produce three scores: Entailment (E), Contradiction (C), and Neutral (N).
- Calibration: A logistic regression model combines the base QA score with the NLI scores (e.g.,
QA+E+C) to produce a final, more robust confidence metric.
Experimental Setup and Key Insights
The study evaluated this approach across 16 datasets, categorized into 9 multiple-choice tasks (such as RACE, SciQ, and CosmosQA) and 7 extractive QA tasks (such as SQuAD and BioASQ). This breadth ensures that the findings are not specific to one domain or question format.
Several key insights emerged from the data:
- Contradiction as a Superior Filter: In several experiments, using the contradiction score (C) alone outperformed the entailment score (E). This indicates that current NLI models are often better at identifying falsehoods than confirming truths.
- The "Safety" Signal: In "Selective QA"—where models can choose to abstain from answering if they aren't confident—the contradiction signal proved exceptionally potent. It acts as a safety net, catching high-confidence hallucinations that entailment-only systems might miss.
- Neutrality is Noise: The "Neutral" score (N) showed little to no correlation with answer accuracy and was eventually excluded from the final optimized models.
Results
The integration of contradiction signals led to measurable improvements in accuracy across the board. In multiple-choice settings, the QA+E+C model reached an average accuracy of 84.57%, surpassing the baseline QA model's 84.08%.
Furthermore, in selective QA tasks focusing on the top 20% most confident answers, the QA+C model achieved a remarkable average accuracy of 97.57%. Statistical analysis confirmed that the contradiction score has a stronger correlation with correctness than the entailment score, particularly in multiple-choice formats.
Conclusion and Future Work
While the contradiction signal is powerful, its implementation faces challenges. Running an NLI model for every candidate answer is computationally expensive, making it difficult to scale to open-domain QA with thousands of passages. Future research should explore "amortizing" these costs, perhaps through NLI-based retrieval techniques.
Additionally, the QA2D process remains a bottleneck; improvements in rephrasing complex questions could further enhance the reliability of the NLI scores. Finally, moving toward more "granular" NLI—where the model identifies the specific reason for a contradiction—could provide even deeper insights into model reasoning and safety.
Summary updated