Abstract
How can prompting a large language model like GPT-3 with explanations improve in-context learning? We focus specifically on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. Including explanations in the prompt and having the model generate them does not consistently improve performance in the settings we study, contrary to recent results on symbolic reasoning tasks (Nye et al., 2021; Wei et al., 2022). Despite careful prompting, explanations generated by GPT-3 may not even be factually grounded in the input, even on simple tasks with straightforward extractive explanations. However, these flawed explanations can still be useful as a way to verify GPT-3's predictions post-hoc. Through analysis in three settings, we show that explanations judged as good by humans—those that are logically consistent with the input and the prediction—usually indicate more accurate predictions. Following these observations, we present a framework for calibrating model predictions based on the reliability of the explanations. Our framework trains calibrators using automatically extracted scores that approximately assess the reliability of explanations, which helps improve performance across three different datasets.
Introduction
In the paradigm of few-shot in-context learning, researchers often employ "Chain-of-Thought" prompting, where Large Language Models (LLMs) are asked to provide step-by-step reasoning before arriving at an answer. The assumption is twofold: first, that this process improves the model's reasoning capabilities, and second, that it provides a transparent window into the model's logic.
However, this study investigates whether these generated explanations are actually trustworthy. We explore two critical dimensions: effectiveness (do explanations actually help?) and reliability (are the explanations grounded in the text and logically sound?). Our findings suggest a significant gap between the perceived and actual utility of these explanations, revealing that models often act as "logical liars"—producing consistent but non-factual reasoning.
Methodology and Datasets
We evaluated four major LLMs—OPT, GPT-3 (davinci), and two versions of InstructGPT (text-davinci-001 and text-davinci-002)—across three prompting strategies: standard Few-Shot, Explain-then-Predict (E-P), and Predict-then-Explain (P-E).
The evaluation utilized three distinct textual reasoning benchmarks:
- SYNTH: A custom synthetic multi-hop QA dataset requiring the linking of two facts.
- Adversarial HotpotQA (ADVHOTPOT): A challenging multi-hop QA dataset featuring "distractor" paragraphs to test robustness.
- E-SNLI: A natural language inference dataset where models must determine entailment or contradiction, supported by human-written explanations.
The Reliability Gap: Consistency vs. Factuality
A key finding is the distinction between consistency and factuality. LLMs are remarkably good at being consistent—their explanations usually lead logically to their chosen answer. However, they frequently fail at being factual, often hallucinating details not present in the source context.
As shown in the figure above, there is a strong correlation between the reliability of an explanation and the accuracy of the prediction. When a model produces a non-factual or inconsistent explanation, the final answer is overwhelmingly likely to be incorrect. This suggests that while explanations might not always improve performance, they serve as a powerful diagnostic signal.
Calibration via Explanation
To leverage this diagnostic signal, we propose a Post-hoc Calibrator. Instead of accepting the model's output at face value, a lightweight linear model acts as a "judge." This calibrator evaluates the model's confidence score alongside a "factuality score" of the explanation, derived from lexical overlap with the source text.
By identifying when a model begins to hallucinate in its reasoning, the calibrator can effectively lower the confidence in the associated prediction, leading to better overall system reliability. On the SYNTH dataset, this approach boosted InstructGPT’s accuracy from 52.4% to 74.8%.
Key Results and Discussion
- Instruction-Tuning as a Prerequisite: Most models saw negligible gains from explanations. However,
text-davinci-002showed a significant jump (e.g., from 72.0% to 86.9% on SYNTH), indicating that specific instruction-tuning is necessary to unlock the power of reasoning prompts. - Hallucination as a Signal: The most profound insight is that non-factuality is a red flag. If a model starts hallucinating details in its explanation, it is highly likely that the final prediction is also incorrect.
Conclusion
While LLM explanations are often factually shaky, their failure modes are predictable. Non-factuality in reasoning is a strong indicator of incorrect answers. By treating explanations as a signal for calibration rather than just a means of generation, we can significantly improve the reliability of LLM-based reasoning systems. Future work should explore more sophisticated entailment models to replace simple lexical overlap as a measure of factuality, addressing the "transparency crisis" inherent in closed-source models.
Summary updated