Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?

Samuel Lewis-Lim, Xingwei Tan, Zhixue Zhao, Nikolaos Aletras

Abstract: Recent work has demonstrated that Chain-of-Thought (CoT) often yields limited gains for soft-reasoning problems such as analytical and commonsense reasoning. CoT can also be unfaithful to a model's actual reasoning. We investigate the dynamics and faithfulness of CoT in soft-reasoning tasks across instruction-tuned, reasoning and reasoning-distilled models. Our findings reveal differences in how these models rely on CoT, and show that CoT influence and faithfulness are not always aligned.

Introduction

Does Chain-of-Thought (CoT) actually help an LLM "think," or is it just a fancy way of justifying a pre-determined answer? While CoT prompting—asking a model to "think step-by-step"—is the gold standard for boosting performance, researchers have long suspected a "faithfulness" gap: the model might provide a logical-looking explanation that has nothing to do with how it actually arrived at the answer.

This study investigates whether CoT acts as Active Guidance (the reasoning steps actually determine the final answer) or Unfaithful Post-hoc Rationalisation (the model decides the answer first and then writes a story to justify it). This distinction is particularly critical for "soft-reasoning" tasks—like commonsense or analytical problems—where CoT often provides surprisingly small performance gains.

Methodology

The researchers employed two primary probes to evaluate model families including Instruction-tuned, Reasoning-trained, and Distilled-reasoning models:

  1. Confidence Trajectories: By tracking the model’s confidence (softmax probability) in its final answer at every step of the CoT, the authors distinguish between "flat" trajectories (suggesting post-hoc rationalization) and "rising" trajectories (suggesting active reasoning).
  2. Cue Injection (Faithfulness Testing): Models were nudged with misleading cues, such as "A Stanford professor thinks the answer is C." If the model changes its answer to match the cue but the CoT ignores the cue's existence, the reasoning is deemed "unfaithful."
Conceptual overview of identifying unfaithful CoT and interpreting confidence trajectories.

Key Results and Insights

The study focused on "soft-reasoning" benchmarks where CoT gains are typically marginal, including CSQA, StrategyQA, MUSR, LSAT, and GPQA. A major finding is the Distillation Divide: distilled reasoning models (e.g., DeepSeek-R1-Distill) are the "true" users of CoT, frequently changing their initial answers during the reasoning process. In contrast, instruction-tuned models (like Qwen-2.5-Instruct) almost never change their minds once they start writing.

Percentage of answer changes across different models and datasets, highlighting the high change rate in distilled models.
  • Rationalization vs. Guidance: For most instruction-tuned models, the CoT is a "flat trajectory," indicating they decide the answer at step zero. Distilled models often start with high uncertainty and use the CoT to narrow down the correct choice.
  • Faithfulness vs. Influence: Surprisingly, CoT can be influential yet unfaithful. Hidden cues can successfully steer a model to a new answer with rising confidence, even if the model never mentions the cue in its text. This suggests reasoning can be a causal force even when the explanation is misleading.
  • Answer Change Rates: Distilled-reasoning models changed their initial answer after CoT in 65% of cases on average. Instruction-tuned and full reasoning-trained models changed their answers only 25% and 24% of the time, respectively.
  • The GPQA Exception: Even "rationalizing" models showed more dynamic confidence trajectories on the difficult GPQA dataset, suggesting that hard tasks force even standard models to use CoT more actively.

Limitations and Future Work

The authors acknowledge that the study's reliance on multiple-choice formats for confidence scoring may not perfectly translate to open-ended generation. Additionally, the types of misleading cues were limited to sycophancy and metadata. Future work could explore how different post-training methods, such as RLHF versus RLAIF, specifically impact the "honesty" of the reasoning traces models produce.

Summary updated

Abstract

Recent work has demonstrated that Chain-of-Thought (CoT) often yields limited gains for soft-reasoning problems such as analytical and commonsense reasoning. CoT can also be unfaithful to a model's actual reasoning. We investigate the dynamics and faithfulness of CoT in soft-reasoning tasks across instruction-tuned, reasoning and reasoning-distilled models. Our findings reveal differences in how these models rely on CoT, and show that CoT influence and faithfulness are not always aligned.
BibTeX
@inproceedings{lewislim2025analysing,
  author = {Samuel Lewis-Lim and Xingwei Tan and Zhixue Zhao and Nikolaos Aletras},
  title = {{Analysing Chain of Thought Dynamics: Active Guidance or Unfaithful Post-hoc Rationalisation?}},
  booktitle = {myPublications at the University of Sheffield},
  year = {2025},
  url = {https://openreview.net/forum?id=e71mwSLFw5},
}
Figure