Abstract: Recent work has demonstrated that Chain-of-Thought (CoT) often yields limited gains for soft-reasoning problems such as analytical and commonsense reasoning. CoT can also be unfaithful to a model's actual reasoning. We investigate the dynamics and faithfulness of CoT in soft-reasoning tasks across instruction-tuned, reasoning and reasoning-distilled models. Our findings reveal differences in how these models rely on CoT, and show that CoT influence and faithfulness are not always aligned.
Introduction
Does Chain-of-Thought (CoT) actually help an LLM "think," or is it just a fancy way of justifying a pre-determined answer? While CoT prompting—asking a model to "think step-by-step"—is the gold standard for boosting performance, researchers have long suspected a "faithfulness" gap: the model might provide a logical-looking explanation that has nothing to do with how it actually arrived at the answer.
This study investigates whether CoT acts as Active Guidance (the reasoning steps actually determine the final answer) or Unfaithful Post-hoc Rationalisation (the model decides the answer first and then writes a story to justify it). This distinction is particularly critical for "soft-reasoning" tasks—like commonsense or analytical problems—where CoT often provides surprisingly small performance gains.
Methodology
The researchers employed two primary probes to evaluate model families including Instruction-tuned, Reasoning-trained, and Distilled-reasoning models:
- Confidence Trajectories: By tracking the model’s confidence (softmax probability) in its final answer at every step of the CoT, the authors distinguish between "flat" trajectories (suggesting post-hoc rationalization) and "rising" trajectories (suggesting active reasoning).
- Cue Injection (Faithfulness Testing): Models were nudged with misleading cues, such as "A Stanford professor thinks the answer is C." If the model changes its answer to match the cue but the CoT ignores the cue's existence, the reasoning is deemed "unfaithful."
Key Results and Insights
The study focused on "soft-reasoning" benchmarks where CoT gains are typically marginal, including CSQA, StrategyQA, MUSR, LSAT, and GPQA. A major finding is the Distillation Divide: distilled reasoning models (e.g., DeepSeek-R1-Distill) are the "true" users of CoT, frequently changing their initial answers during the reasoning process. In contrast, instruction-tuned models (like Qwen-2.5-Instruct) almost never change their minds once they start writing.
- Rationalization vs. Guidance: For most instruction-tuned models, the CoT is a "flat trajectory," indicating they decide the answer at step zero. Distilled models often start with high uncertainty and use the CoT to narrow down the correct choice.
- Faithfulness vs. Influence: Surprisingly, CoT can be influential yet unfaithful. Hidden cues can successfully steer a model to a new answer with rising confidence, even if the model never mentions the cue in its text. This suggests reasoning can be a causal force even when the explanation is misleading.
- Answer Change Rates: Distilled-reasoning models changed their initial answer after CoT in 65% of cases on average. Instruction-tuned and full reasoning-trained models changed their answers only 25% and 24% of the time, respectively.
- The GPQA Exception: Even "rationalizing" models showed more dynamic confidence trajectories on the difficult GPQA dataset, suggesting that hard tasks force even standard models to use CoT more actively.
Limitations and Future Work
The authors acknowledge that the study's reliance on multiple-choice formats for confidence scoring may not perfectly translate to open-ended generation. Additionally, the types of misleading cues were limited to sycophancy and metadata. Future work could explore how different post-training methods, such as RLHF versus RLAIF, specifically impact the "honesty" of the reasoning traces models produce.
Summary updated