Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning

Oyvind Tafjord, Bhavana Dalvi Mishra, Peter Clark

TL;DR

This paper introduces "Entailer," a novel question-answering system that generates not just answers but also "faithful and truthful" chains of reasoning. These chains explain how an answer is derived from the model's own internal beliefs, offering unprecedented transparency and enabling better debugging and understanding of AI reasoning.

What Problem Does This Paper Address?

Current question-answering (QA) systems, even those using large language models (LLMs), often provide answers without clear explanations. When explanations are generated (e.g., "chains of thought"), they aren't always "faithful" (the answer doesn't logically follow from the explanation) or "truthful" (the model doesn't actually "believe" the steps in its own explanation). This lack of transparency makes it hard to trust the system, diagnose errors, or understand its underlying knowledge. Entailer aims to bridge this gap by generating reasoning chains that are both logically sound and reflective of the model's internal state.

Methods

Entailer employs a unique "over-generate and verify" approach. It uses a backward-chaining model to recursively generate potential premises that could lead to an answer hypothesis. Crucially, it then uses a "self-querying" mechanism to verify if the model itself "believes" these premises and the entailment steps. This verification process ensures that the generated reasoning is not only logically valid but also grounded in the model's own internal knowledge. The system iteratively builds these chains, pruning branches that don't improve confidence, to find the best supporting proof for an answer.

Datasets

  • EntailmentBank: Used for training, this dataset contains science questions with multi-step entailment trees. It was augmented with a new crowdsourced dataset to include negative entailments (where premises and reasoning are incorrect).
  • OBQA (OpenBookQA) and QuaRTz: These datasets were used for evaluating Entailer's QA accuracy in a zero-shot setting.

Key Insights and Ideas

  • "Faithful and Truthful" Reasoning: The core innovation is the emphasis on reasoning that is both logically sound (faithful) and reflects the model's internal beliefs (truthful). This moves beyond simply generating plausible-sounding explanations.
  • Self-Verification as a Belief Check: The system's ability to query itself about the truthfulness of premises and entailment steps is a key mechanism for ensuring the "truthfulness" of the reasoning.
  • Transparency for Debugging: By materializing the model's beliefs, Entailer provides a direct window into its decision-making process, making it easier to identify and correct errors stemming from faulty beliefs or reasoning.
  • Reasoning Depth vs. Accuracy: The study found that while deeper proofs can offer more information, they don't significantly improve accuracy over shallower ones, suggesting a trade-off between explanation depth and computational cost.

Key Results

  • Competitive QA Accuracy: Entailer achieved QA accuracy comparable to direct QA systems (around 75% on OBQA and QuaRTz) without significant performance loss, demonstrating that generating proofs doesn't hinder answer correctness.
  • Human Judged Quality: In human evaluations, Entailer's proofs were judged to clearly follow from premises in over 70% of cases, significantly outperforming a strong baseline (Macaw) at 34%. Entailer's explanations were also preferred by annotators (57% vs. 23%).
  • Premise Correctness: Approximately 90% of Entailer's self-verified premises were judged correct by users, indicating high reliability in its internal belief checks.
  • Failure Analysis: Failures were categorized into incorrect beliefs (33%), dataset errors (20%), and reasoning errors (47%), providing insights into the types of errors the system makes.

Limitations and Future Work

  • Proof Coherence: Entailer can still produce invalid or near-tautologous entailments, suggesting room for improvement in proof generation and scoring techniques.
  • Belief Consistency: The system assumes consistent beliefs, but real-world models can be inconsistent, requiring exploration of global belief notions.
  • Computational Cost: Recursive proof generation is computationally expensive, necessitating research into faster search algorithms.
  • Teachable Systems: The authors envision Entailer as a foundation for "teachable systems" where users can correct faulty beliefs, leading to interactive learning and improved AI over time. This is an exciting avenue for future work.

Summary updated

Abstract

Our goal is a question-answering (QA) system that can show how its answers are implied by its own internal beliefs via a systematic chain of reasoning. Such a capability would allow better understanding of why a model produced the answer it did. Our approach is to recursively combine a trained backward-chaining model, capable of generating a set of premises entailing an answer hypothesis, with a verifier that checks that the model itself believes those premises (and the entailment itself) through self-querying. To our knowledge, this is the first system to generate multistep chains that are both faithful (the answer follows from the reasoning) and truthful (the chain reflects the system's own internal beliefs). In evaluation using two different datasets, users judge that a majority (70%+) of generated chains clearly show how an answer follows from a set of facts - substantially better than a high-performance baseline - while preserving answer accuracy. By materializing model beliefs that systematically support an answer, new opportunities arise for understanding the model's system of belief, and diagnosing and correcting its misunderstandings when an answer is wrong.
BibTeX
@inproceedings{tafjord2022entailer,
  author = {Oyvind Tafjord and Bhavana Dalvi Mishra and Peter Clark},
  title = {{Entailer: Answering Questions with Faithful and Truthful Chains of Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=w5e8M0O2ehq},
}
Figure