Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation

Yueer Pan, Yuexi PanarXiv 2026 · arXiv

TL;DR

This paper demonstrates that evaluating Chain-of-Thought (CoT) faithfulness only on correctly answered questions introduces a significant selection bias, leading to inflated scores. By formalizing this bias through a causal framework, the authors show that "correct-only" evaluations can overestimate a model's true faithfulness by over 7 percentage points.

What Problem Does This Paper Address?

Large Language Models (LLMs) often provide a "Chain-of-Thought" (CoT)—a step-by-step explanation of their reasoning. A key question for researchers is faithfulness: does the model actually use the reasoning steps it shows us, or is the CoT just a post-hoc justification for an answer it reached through some other "shortcut"?

To measure this, researchers often intervene on a reasoning step (e.g., changing a number in an intermediate calculation) and see if the final answer updates accordingly. However, many studies only perform this analysis on instances where the model originally gave a correct answer. The assumption is that incorrect answers are "broken" anyway, so their faithfulness doesn't matter.

This paper argues that this "correctness conditioning" fundamentally changes the population being measured. It creates a "selection gap" where the reported faithfulness score is higher than the model's actual faithfulness across all tasks.

Figure 1: CoTsel overview. Conceptual task intervention and correctness selection (top), with measured faithfulness and selection gaps on arithmetic DAGs (bottom).

Methods

The authors introduce CoTsel, a framework for diagnosing selection bias in CoT evaluation. Their approach includes:

  1. A Causal Framework: They model the reasoning process using a seven-variable Structural Causal Model (SCM). This allows them to mathematically derive the "selection gap" ($\Delta$) as a function of the covariance between answer correctness and intervention compliance.
  2. Intervention Protocol: They use a "counterfactual oracle" to test faithfulness. For a given task, they:
  3. Record the model's original (factual) answer.
  4. Intervene by changing one intermediate step in the reasoning chain.
  5. Check if the model's new answer matches the "counterfactual oracle"—the mathematically correct answer given the intervention.
  6. Gap Decomposition: They use "operation count" (the number of steps in a task) as a proxy for difficulty to see how much of the selection gap is simply due to the model being more faithful on easier tasks.
Figure 2: Correct factual answers can conceal different intervention responses. The two rules agree before the edit and diverge afterward.

Datasets

The study focuses on procedural tasks where the ground truth is mathematically certain, allowing for a perfect "counterfactual oracle":

  • Arithmetic DAGs: Integer operations with varying depths (2–4).
  • Boolean Circuits: Logical gates (AND, OR, NOT) with depths of 2–8.
  • Graph Reachability: Determining if a path exists between nodes in graphs with 5–12 nodes.

These datasets are crucial because, unlike natural language tasks, they allow researchers to precisely calculate what the "faithful" answer should be after any arbitrary change to the reasoning chain.

Key Insights and Ideas

  • The Selection Gap is Real: Filtering for correct answers consistently inflates faithfulness scores. In their primary test with the GLM-5.3 model, the faithfulness score jumped from 50.7% (all traces) to 58.1% (correct only).
  • Difficulty Confounding: A major driver of this bias is that easier tasks are "confounders." On easy tasks, a model is more likely to be correct and more likely to be faithful. By selecting only correct answers, you are disproportionately selecting easy tasks, which makes the model look more faithful than it is on average.
  • Collider Bias: The paper identifies a more subtle bias: conditioning on correctness can create a spurious association between CoT reliance and "shortcuts." If a model can get an answer right either by reasoning or by using a shortcut, then knowing it got the answer right makes it less likely it was actually reasoning faithfully if shortcuts are available.
  • Protocol Inconsistency: A review of 18 major CoT papers found a wide variety of selection rules, making it difficult to compare faithfulness results across different studies.
Figure 3: Structural causal model before and after correctness conditioning.

Key Results

  • Quantifiable Inflation: Across four different model deployments (GLM and DeepSeek variants), the authors observed positive selection gaps ranging from +1.6 to +7.4 percentage points.
  • Statistical Significance: The primary GLM-5.3 run showed a highly significant gap ($p < 0.0001$). A pooled analysis across all models yielded an Odds Ratio of 1.70, meaning correctly answered instances were 1.7 times more likely to be "faithful" than the general population.
  • Attribution: Using their decomposition method, they found that task difficulty (operation count) accounted for about 42.8% of the gap in the GLM-5.3 model, leaving the majority of the bias still "unattributed" to simple difficulty metrics.

Limitations and Future Work

  • Generalization to Natural Language: The study relies on procedural tasks with deterministic oracles. In natural language CoT (e.g., "Why is the sky blue?"), there isn't always a single "correct" answer after an intervention, making it much harder to measure the selection gap.
  • Shortcut Detection: While the theory predicts "collider bias" from shortcuts, the authors' main experiments used "neutral framing" where models rarely used shortcuts. They had to use adversarial "probes" to elicit shortcut behavior, suggesting that while the bias exists in theory, its real-world impact depends heavily on how much the model relies on cues versus reasoning.
  • Future Directions: The authors suggest that future faithfulness evaluations should report results for all traces whenever possible, or at least explicitly state their correctness filtering criteria to allow for better cross-study comparisons.

Summary updated

Abstract

Restricting chain-of-thought (CoT) faithfulness evaluation to correctly answered instances changes the population being measured. We study this selection step in a seven-variable structural causal model and derive the exact observed gap $Δ= \operatorname{Cov}(A, C)/P(C=1)$. The model distinguishes difficulty confounding from shortcut-induced collider bias and states the conditional-independence assumption needed to express the gap through latent response propensities. A protocol review of 18 papers distinguishes answer-correctness restrictions from agreement-based selection, benchmark construction, and training-data filtering. We evaluate four sub-ceiling deployments on arithmetic tasks and observe positive selection gaps in all four. The primary GLM-5.3 run yields a +7.4 percentage-point gap ($n = 1000$, $p < 0.0001$), and pooled Cochran-Mantel-Haenszel analysis gives OR = 1.70 ($p < 10^{-6}$). An assumption-free covariance decomposition assigns 42.8% of the GLM-5.3 gap to between-stratum variation in interventionable-operation count, leaving the majority unattributed. When counterfactual compliance is observable for every instance, evaluating all traces directly estimates population faithfulness without correctness filtering.
BibTeX
@misc{pan2026correctness,
  author = {Yueer Pan and Yuexi Pan},
  title = {{Correctness Conditioning as Selection in Chain-of-Thought Faithfulness Evaluation}},
  year = {2026},
  eprint = {2610.37070},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37070},
}
Figure