TL;DR
This paper provides a deep dive into why Chain-of-Thought (CoT) prompting works—and why it sometimes "lies." By analyzing information flow and difficulty, the authors reveal that unfaithful reasoning often occurs because models bypass their own flawed CoTs to retrieve correct information directly from the prompt. They introduce QUIRE, a novel algorithm that improves both reasoning accuracy and faithfulness by explicitly recalling missing context and weighting rationales by their information gain.
What Problem Does This Paper Address?
Chain-of-Thought (CoT) has become the gold standard for eliciting reasoning from Large Language Models (LLMs). However, its success is inconsistent: it excels at math but can be redundant or even harmful for commonsense tasks.
More concerning is the issue of unfaithfulness: a model might provide a perfectly correct answer preceded by a completely nonsensical or incorrect reasoning chain. This suggests the model isn't actually "thinking" through the steps it shows us. The authors set out to understand the underlying patterns of CoT effectiveness and explain the mechanics behind this unfaithful behavior.
Methods
The researchers employed several sophisticated lenses to analyze model internals:
- Information Gain (IG): A metric based on information theory to quantify how much "new" information a CoT adds beyond what is already present in the question.
- Integrated Gradient Attribution (IGA): A technique used to trace "information flow," measuring how much each token in a CoT actually influences the final answer.
- QUIRE (QUestion Information Recall and Enhancement): A two-step algorithm designed to fix unfaithful CoT.
- AAE Recall: It first generates a "raw" answer to identify which parts of the original question are most relevant (using attribution). These are then fed back to the model as "hints" to generate a more grounded CoT.
- IG Vote: It generates multiple reasoning paths and uses Information Gain as a weight to vote for the final answer, favoring rationales that provide more substantial reasoning.
Datasets
The study evaluated four advanced open-source models (Mistral-7B, Gemma2-9B, Llama3.1-8B, and Qwen2.5-14B) across nine representative benchmarks:
- Mathematical Reasoning: GSM8K, GSMIC, AQuA.
- Logical Reasoning: ProofWriter, FOLIO, ProntoQA.
- Commonsense Reasoning: WinoGrande, SocialIQA, ECQA.
Key Insights and Ideas
-
CoT is a "Hard Problem" Specialist: CoT provides minimal gains (or even degrades performance) on easy questions. Its true power is only unlocked as problem difficulty increases.
Figure 2: Performance on different problem difficulty levels with and without CoT prompting (Llama3.1-8B). -
The "Bypass" Mechanism of Unfaithfulness: The authors discovered that when a CoT is unfaithful (wrong reasoning, right answer), the model is often "recalling" the correct information directly from the original question during the final answer step, effectively ignoring its own flawed reasoning chain.
-
Monotonicity Matters: In effective CoT, the influence of the reasoning steps on the final answer should steadily increase as the chain progresses. If the information flow is flat, the CoT is likely not helping.
Figure 5: Information flow between the CoT and an- swer. ‘Step’ indicates sequential positions within the CoT, where 0 is the beginning and 100 is the end. -
Information Gain Trade-off: Mathematical tasks have low Information Gain from the question (meaning the model has to generate a lot of new logic), while commonsense tasks have high IG (meaning the answer is often already "hidden" in the prompt). CoT is most effective when it forces the model to generate information not explicitly in the prompt.
Key Results
- Performance Boost: The proposed QUIRE method improved reasoning accuracy by up to 2.4% on the ProofWriter dataset.
- Faithfulness Improvement: More significantly, QUIRE increased "Faithful BertScore" (a metric measuring the alignment between reasoning quality and answer correctness) by up to 5.6% on ProntoQA.
- Cross-Model Consistency: The findings regarding difficulty and information flow held true across different model families (Llama, Gemma, Mistral, Qwen), suggesting these are fundamental properties of how current LLMs handle CoT.
Limitations and Future Work
- White-Box Dependency: The analysis relies on gradient information, which is unavailable for "black-box" proprietary models like GPT-4 or Claude. Future work is needed to adapt these insights for API-based models.
- Empirical vs. Theoretical: While the paper provides strong empirical evidence that improving faithfulness boosts effectiveness, a formal theoretical proof of this relationship remains an open challenge.
- Computational Cost: Methods like QUIRE involve multiple generation passes and attribution steps, which increases the computational overhead compared to standard CoT.
Summary updated