Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning

Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark

Abstract

Large language models (LLMs) often provide answers without clear, logical justifications, leading to issues of unfaithfulness and untruthfulness in their explanations. We introduce Entailer, a question-answering system that generates multi-step reasoning chains by combining recursive backward-chaining with a self-verification loop. Entailer ensures that its explanations are faithful to its internal logic and truthful to its underlying beliefs. Our results show that Entailer significantly improves human interpretability without sacrificing predictive accuracy, achieving over 70% human preference for its reasoning chains compared to strong baselines.

Introduction

As large language models (LLMs) become integrated into critical decision-making processes, the need for explainability becomes paramount. However, standard techniques like "Chain of Thought" (CoT) prompting often produce explanations that are either unfaithful—where the logic does not actually drive the model's answer—or untruthful—where the model cites facts it would otherwise reject as false.

Entailer addresses these shortcomings by operationalizing the concept of "belief." In this framework, a model is considered to believe a statement if it consistently affirms its truth when queried directly. By forcing the model to build answers upon these internal beliefs through a structured reasoning process, Entailer transforms the LLM from a "black box" into a transparent, inspectable system.

The Entailer Framework

Entailer utilizes a T5-11B model trained to operate across three functional "angles":

  • Generation ($H \to P$): Given a target hypothesis, the model generates a set of premises intended to entail it.
  • Direct Verification ($H \to S_d$): A "fast thinking" component that scores the model's inherent belief in a statement's truth.
  • Entailment Verification ($PH \to S_e$): A "slow thinking" component that evaluates whether a specific set of premises logically leads to the hypothesis.
The Entailer workflow: from hypothesis enumeration to final proof selection.

The system employs a recursive backward-chaining search. It begins with candidate answers and works backward, asking, "What would need to be true for this to be the correct answer?" It iteratively generates and verifies premises, maintaining a self-querying filter that only keeps steps where the model confirms both the truth of the premises and the validity of the logic. The final answer is selected based on the highest-scoring, fully verified proof tree.

Datasets and Training

Entailer was trained on EntailmentBank, a dataset of 1,840 science questions paired with expert-authored entailment trees. To improve the model's robustness, the authors augmented this with a new crowdsourced dataset of multi-premise entailments, doubling the training size. Crucially, this new data included negative examples—flawed proofs for incorrect answers—which helped the model learn to identify and reject invalid logic. The system's performance was evaluated zero-shot on the OBQA (OpenBookQA) and QuaRTz benchmarks, which require complex, multi-hop reasoning.

Results and Evaluation

A key finding of this work is the absence of an "interpretability tax." Entailer achieved a reasoning-based accuracy of 75.2% on OBQA, which is virtually identical to its direct "black box" accuracy. This suggests that structured reasoning can be implemented without degrading the model's core capabilities.

Examples of reasoning chains generated by Entailer for various science questions.

Human evaluation further validates the system's performance:

  • Human Preference: In blind comparisons, human judges found that over 70% of Entailer’s reasoning chains clearly demonstrated how an answer followed from established facts. This is more than double the performance of the high-quality Macaw baseline (34%).
  • Truthfulness: Approximately 90% of the premises generated and self-verified by Entailer were judged as factually correct by humans.
  • Faithfulness: Because Entailer is architecturally constrained to only select answers for which it can find a valid proof, the resulting explanation is inherently linked to the final answer.

Discussion and Future Work

Despite its successes, Entailer faces several challenges. The recursive backward-chaining process is computationally expensive; generating a depth-3 proof for a standard multiple-choice question can take several minutes on high-end hardware. Additionally, the model occasionally produces "near-tautologies" or abductive errors where the logic is circular or weak.

Future research will focus on moving toward a "global notion of belief," ensuring the model maintains a consistent knowledge base across all interactions. Furthermore, the explicit nature of Entailer’s reasoning chains makes it a prime candidate for "Teachable Systems." By allowing users to identify and correct specific false premises within a chain, we can effectively "teach" the model, leading to more reliable and interactive AI assistants.

Summary updated

Abstract

Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would allow better understanding of why a model produced the answer it did. Our approach is to recursively combine a trained backward-chaining model, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises (and the entailment itself) through self-querying. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system's own internal beliefs). In evaluation using two different datasets, users judge that a majority (70%+) of generated chains clearly show how an answer follows from a set of facts - substantially better than a high-performance baseline - while preserving answer accuracy. By materializing model beliefs that systematically support an answer, new opportunities arise for understanding the model's system of belief, and diagnosing and correcting its misunderstandings when an answer is wrong.
BibTeX
@inproceedings{tafjord2022entailer,
  author = {Oyvind Tafjord and Bhavana Dalvi Mishra and Peter Clark},
  title = {{Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=w5e8M0O2ehq},
}
Figure