The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning

Xi Ye, Greg DurrettNeurIPS 2022 · OpenReview

TL;DR

Adding human-like explanations to Large Language Model (LLM) prompts doesn't always improve their reasoning—and the explanations they generate are often factually incorrect. However, this paper demonstrates that the "factuality" of a model's explanation is a powerful signal for its overall accuracy, allowing researchers to build a post-hoc calibrator that significantly boosts performance.

What Problem Does This Paper Address?

In the world of "Chain-of-Thought" prompting, the prevailing wisdom is that asking a model to "show its work" leads to better answers. While this holds true for math and symbolic logic, the authors of this paper ask: does it work for general textual reasoning, like question answering (QA) and natural language inference (NLI)?

The challenge is twofold. First, standard LLMs often fail to see performance gains from these added explanations. Second, even when a model provides an explanation, it might be a "hallucination"—a convincing-sounding justification that isn't actually grounded in the provided text. The authors set out to measure the reliability of these explanations and find a way to use them even when they are flawed.

Methods

The researchers compared three prompting strategies across four LLMs (OPT, GPT-3, and two versions of InstructGPT):

  1. Standard Few-Shot: Providing input-label pairs.
  2. Explain-then-Predict (E-P): The model generates an explanation before the answer.
  3. Predict-then-Explain (P-E): The model generates the answer before the explanation.
Overview of the post-hoc calibrator pipeline

To address the unreliability of these outputs, they introduced a post-hoc calibrator. Instead of blindly trusting the model's top prediction, they trained a simple linear model to adjust the prediction's probability based on a "factuality score." This score was calculated using lexical overlap (how much the explanation's words matched the source text). If an explanation wasn't grounded in the context, the calibrator would lower the confidence in that answer.

Datasets

The study utilized three distinct datasets to test different reasoning flavors:

  • SYNTH: A new, highly controlled synthetic multi-hop QA dataset designed to make it easy to verify if an explanation is factually grounded.
  • ADVHOTPOT: An adversarial version of the HotpotQA dataset. It includes "distractor" paragraphs to trick the model, making reasoning much harder.
  • E-SNLI: A standard Natural Language Inference dataset where explanations are more abstract and human-written, rather than just extracted from the text.

Key Insights and Ideas

Distribution of correct and incorrect predictions based on explanation reliability
  • Consistency $\neq$ Factuality: Models are surprisingly good at making their explanations match their answers (consistency), but they are much worse at making those explanations match the source text (factuality). They can be "consistently wrong."
  • The "Red Flag" Signal: A non-factual explanation is a massive indicator that the prediction is incorrect. Even if the explanation itself is a mess, the mere fact that it's "unreliable" provides a signal that can be used to filter out bad answers.
  • The text-davinci-002 Outlier: Most models (OPT, GPT-3, text-davinci-001) showed only marginal gains from explanations. However, text-davinci-002 showed a significant "jump" in ability, suggesting that specific instruction-tuning or RLHF (Reinforcement Learning from Human Feedback) might be the key to unlocking explanation-based reasoning.
  • Calibration over Prompting: Simply "plugging in" explanations into a prompt is hit-or-miss. Using those explanations as a diagnostic tool for calibration is a much more robust way to improve accuracy.

Key Results

  • Accuracy Boost via Calibration: On the synthetic dataset, using a factuality check to reject bad answers boosted accuracy from 52.4% to 74.8%.
  • E-SNLI Improvements: On the more complex E-SNLI task, the explanation-based calibrator (P-E+EXPLCAL) achieved 68.5% accuracy, significantly outperforming standard few-shot learning (56.8%).
  • Reliability as a Proxy: On the adversarial dataset (ADVHOTPOT), the correlation between an explanation's factuality and the answer's correctness was 80%, far higher than the model's raw accuracy alone.

Limitations and Future Work

The authors acknowledge that their "factuality score" relies on lexical overlap (word matching), which is a relatively "weak" signal. It doesn't capture deep semantic meaning or complex logical entailment. A model could technically use different words to describe a fact and be penalized unfairly, or use the right words in a nonsensical order and be rewarded.

Future work could involve using more sophisticated "entailment models" to judge explanations. Additionally, because text-davinci-002 performed so much better than its predecessors, there is a need for more transparency regarding how these models are trained to understand why some "get" explanations while others don't.

Summary updated

Abstract

Does prompting a large language model (LLM) like GPT-3 with explanations improve in-context learning? We study this question on two NLP tasks that involve reasoning over text, namely question answering and natural language inference. We test the performance of four LLMs on three textual reasoning datasets using prompts that include explanations in multiple different styles. For these tasks, we find that including explanations in the prompts for OPT, GPT-3 (davinci), and InstructGPT (text-davinci-001) only yields small to moderate accuracy improvements over standard few-show learning. However, text-davinci-002 is able to benefit more substantially.We further show that explanations generated by the LLMs may not entail the models’ predictions nor be factually grounded in the input, even on simple tasks with extractive explanations. However, these flawed explanations can still be useful as a way to verify LLMs’ predictions post-hoc. Through analysis in our three settings, we show that explanations judged by humans to be good—logically consistent with the input and the prediction—more likely cooccur with accurate predictions. Following these observations, we train calibrators using automatically extracted scores that assess the reliability of explanations, allowing us to improve performance post-hoc across all of our datasets.
BibTeX
@inproceedings{ye2022unreliability,
  author = {Xi Ye and Greg Durrett},
  title = {{The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=s_tbKY4cXL},
}
Figure