TL;DR
Adding human-like explanations to Large Language Model (LLM) prompts doesn't always improve their reasoning—and the explanations they generate are often factually incorrect. However, this paper demonstrates that the "factuality" of a model's explanation is a powerful signal for its overall accuracy, allowing researchers to build a post-hoc calibrator that significantly boosts performance.
What Problem Does This Paper Address?
In the world of "Chain-of-Thought" prompting, the prevailing wisdom is that asking a model to "show its work" leads to better answers. While this holds true for math and symbolic logic, the authors of this paper ask: does it work for general textual reasoning, like question answering (QA) and natural language inference (NLI)?
The challenge is twofold. First, standard LLMs often fail to see performance gains from these added explanations. Second, even when a model provides an explanation, it might be a "hallucination"—a convincing-sounding justification that isn't actually grounded in the provided text. The authors set out to measure the reliability of these explanations and find a way to use them even when they are flawed.
Methods
The researchers compared three prompting strategies across four LLMs (OPT, GPT-3, and two versions of InstructGPT):
- Standard Few-Shot: Providing input-label pairs.
- Explain-then-Predict (E-P): The model generates an explanation before the answer.
- Predict-then-Explain (P-E): The model generates the answer before the explanation.
To address the unreliability of these outputs, they introduced a post-hoc calibrator. Instead of blindly trusting the model's top prediction, they trained a simple linear model to adjust the prediction's probability based on a "factuality score." This score was calculated using lexical overlap (how much the explanation's words matched the source text). If an explanation wasn't grounded in the context, the calibrator would lower the confidence in that answer.
Datasets
The study utilized three distinct datasets to test different reasoning flavors:
- SYNTH: A new, highly controlled synthetic multi-hop QA dataset designed to make it easy to verify if an explanation is factually grounded.
- ADVHOTPOT: An adversarial version of the HotpotQA dataset. It includes "distractor" paragraphs to trick the model, making reasoning much harder.
- E-SNLI: A standard Natural Language Inference dataset where explanations are more abstract and human-written, rather than just extracted from the text.
Key Insights and Ideas
- Consistency $\neq$ Factuality: Models are surprisingly good at making their explanations match their answers (consistency), but they are much worse at making those explanations match the source text (factuality). They can be "consistently wrong."
- The "Red Flag" Signal: A non-factual explanation is a massive indicator that the prediction is incorrect. Even if the explanation itself is a mess, the mere fact that it's "unreliable" provides a signal that can be used to filter out bad answers.
- The
text-davinci-002Outlier: Most models (OPT, GPT-3,text-davinci-001) showed only marginal gains from explanations. However,text-davinci-002showed a significant "jump" in ability, suggesting that specific instruction-tuning or RLHF (Reinforcement Learning from Human Feedback) might be the key to unlocking explanation-based reasoning. - Calibration over Prompting: Simply "plugging in" explanations into a prompt is hit-or-miss. Using those explanations as a diagnostic tool for calibration is a much more robust way to improve accuracy.
Key Results
- Accuracy Boost via Calibration: On the synthetic dataset, using a factuality check to reject bad answers boosted accuracy from 52.4% to 74.8%.
- E-SNLI Improvements: On the more complex E-SNLI task, the explanation-based calibrator (
P-E+EXPLCAL) achieved 68.5% accuracy, significantly outperforming standard few-shot learning (56.8%). - Reliability as a Proxy: On the adversarial dataset (ADVHOTPOT), the correlation between an explanation's factuality and the answer's correctness was 80%, far higher than the model's raw accuracy alone.
Limitations and Future Work
The authors acknowledge that their "factuality score" relies on lexical overlap (word matching), which is a relatively "weak" signal. It doesn't capture deep semantic meaning or complex logical entailment. A model could technically use different words to describe a fact and be penalized unfairly, or use the right words in a nonsensical order and be rewarded.
Future work could involve using more sophisticated "entailment models" to judge explanations. Additionally, because text-davinci-002 performed so much better than its predecessors, there is a need for more transparency regarding how these models are trained to understand why some "get" explanations while others don't.
Summary updated