TL;DR
This paper investigates how Large Language Models (LLMs) use Chain-of-Thought (CoT) reasoning, specifically whether it actively guides the model to the correct answer or merely rationalizes a pre-determined one. They find that distilled-reasoning models heavily rely on CoT for performance, while instruction-tuned and reasoning models use it less, suggesting different roles for CoT across model types.
What Problem Does This Paper Address?
Chain-of-Thought (CoT) prompting has become a popular technique to improve the reasoning capabilities of LLMs. However, recent research suggests that CoT doesn't always lead to performance gains, especially in "soft-reasoning" tasks (like commonsense reasoning). A key question arises: is CoT truly helping the model reason, or is it just a post-hoc justification for an answer the model already decided on? This paper aims to disentangle these two possibilities by analyzing the "dynamics" of CoT – how the model's confidence changes during the reasoning process – and its "faithfulness" – whether the generated reasoning accurately reflects the model's decision-making.
Methods
The authors employ two primary methods to analyze CoT:
- Confidence Trajectories: They track the model's confidence in its final answer as each step of the CoT is generated. A "rising trajectory" suggests CoT is actively guiding the model, while a "flat trajectory" indicates CoT might be a post-hoc rationalization.
- Faithfulness Testing: They inject misleading "cues" into the prompt (e.g., a fabricated Stanford professor's opinion or metadata) and observe if the model's answer changes. They then check if the CoT explicitly acknowledges using this cue. If the answer changes due to the cue but the CoT doesn't mention it, the CoT is considered "unfaithful."
Datasets
The study uses several existing datasets for evaluation, focusing on those where CoT has shown limited benefit:
- Commonsense Reasoning: CSQA, StrategyQA, MUSR
- Analytical/Reasoning: LSAT, GPQA
Key Insights and Ideas
- Model Type Matters: Different types of LLMs (instruction-tuned, reasoning, distilled-reasoning) exhibit distinct CoT behaviors.
- Distilled Models Live by CoT: Distilled-reasoning models show a high rate of answer changes after CoT and rely heavily on it for good performance, suggesting it's crucial for their reasoning process.
- Instruction-Tuned Models are More Static: These models change their answers less often after CoT, pointing towards CoT acting more as a justification.
- Faithfulness vs. Influence is Not Aligned: A CoT can be unfaithful (not acknowledging a cue) yet still causally influence the answer. Conversely, a faithful CoT might not always influence the final answer. This challenges simple definitions of faithfulness.
- Confidence Trajectories Reveal Dynamics: Flat confidence trajectories often correlate with post-hoc rationalization, while rising trajectories suggest active guidance.
Key Results
- Answer Changes: Distilled-reasoning models changed their initial answers in an average of 65% of cases across datasets, significantly more than instruction-tuned (25%) and reasoning models (24%).
- CoT Gain: Distilled models often showed substantial accuracy improvements with CoT, while instruction-tuned and reasoning models showed smaller or even negative gains on some tasks.
- Faithfulness Scores: Faithfulness scores (proportion of examples where the model changed its answer to match a cue and explicitly verbalized using it) varied significantly. For instance, Qwen3-32B showed higher faithfulness scores (e.g., 0.68 for CSQA with metadata cue) compared to instruction-tuned models (e.g., 0.00 for CSQA with metadata cue).
Limitations and Future Work
- Limited Cues: The faithfulness analysis focused on specific types of cues (metadata and professor cues). It's unknown how unfaithful CoT might manifest in more natural, "in-the-wild" scenarios.
- Multiple-Choice Tasks: The study primarily used multiple-choice tasks. Extending the analysis to long-form generation and planning tasks would provide a broader understanding.
- Future Work: The authors suggest exploring how different post-training methods (like RLHF) affect CoT faithfulness and reliance, and how these dynamics generalize to more complex tasks. Understanding the interplay between influence and faithfulness remains a key area for future research.
Summary updated