Blinded Step Checks for Label-Free Reasoning Self-Training

When a language model trains on its own answers to unlabeled problems, it can teach itself its own mistakes. Current methods guard against this by keeping only answers the model produces consistently, but consistency only removes random slips. It does not remove errors the model makes reliably. There is evidence that many of these reliable errors are rationalizations: the model settles on an answer early and then writes reasoning that justifies it. We propose a check that a rationalizing model cannot pass. Each intermediate claim in a self-generated solution is posed back to the same model as a small standalone question, in a fresh context that never shows the final answer or the rest of the chain. A solution enters the training pool only if its steps hold up when the model cannot see where they were leading.

In two sentences: majority-vote self-training reinforces whatever the model believes most strongly, including beliefs it reached by justifying a hunch. Re-asking each step "blind" removes the hunch from the context, so steps that only made sense as justification tend to fail. That gives a label-free filter aimed at the coherent errors voting cannot catch.

Where the gap is

Three findings motivate the proposal. Unveiling Confirmation Bias in Chain-of-Thought Reasoning measures how strongly a model already favors one answer before it reasons, using the entropy of its direct-answer distribution. When that prior is strong, rationales become shorter and more superficial, and models ignore what their own rationales imply when it conflicts with the hunch. The study is inference-only and never asks what happens when such rationales become training data. Consistency Training Can Entrench Misalignment shows that self-bootstrapping removes brittle behaviors, those that vary across samples, but amplifies coherent ones, and that this gets worse at 70B scale. Its evidence comes from alignment model organisms, not deduction. Unsupervised Post-Training of Foundation Models: A Survey names "false-popular modes" as the main failure of vote-based methods such as TTRL: wrong answers the model happens to sample often. Taken together, these suggest that some of the coherent errors self-training entrenches in reasoning are rationalizations of a prior. If so, they should be detectable by removing the prior's target from the context.

The closest methods fall short in specific ways. CORE-PO (Self-Training Large Language Models with Confident Reasoning) scores each statement of a chain with the model's own probability that it is true, but it does so with the full chain, and therefore the answer, in view. That is exactly the setting where the confirmation-bias study finds judgments bend toward the hunch. Co-rewarding (Co-rewarding) rewards matching final answers across a question and its rephrasing, or between the policy and a slowly updated copy of itself. It never inspects steps, and the slow copy inherits the policy's misconceptions. ReST-MCTS* (ReST-MCTS*) assigns step credit from rollouts, but it needs ground-truth final answers. The proposal borrows its mechanism from two inference-time results. Divide-and-conquer prompting (When Splitting Makes Stronger) shows that clean, isolated contexts stop errors from compounding. ConsisEval (Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?) shows that a model's answer to a hard problem and its answer to the easier subproblem can disagree. We turn that disagreement into a training-time signal.

Core Claim

Blind re-checking of intermediate claims separates correct from incorrect majority pseudo-labels better than vote margin or in-context self-assessment, and the advantage concentrates on questions where the model's prior is strong. Filtering or rewarding self-training data with this signal keeps pseudo-labels precise over iterations where vote-based self-training drifts. The claim is deliberately narrow. Blinding should catch errors driven by the model's hunch about the answer. It should not catch errors that come from a genuinely mistaken rule, because the model will reproduce a mistaken rule blind as well. The experiments must measure how much of the false-popular-mode problem falls into each category.

Research Program

For each unlabeled question we sample $N$ reasoning chains and form the usual majority pseudo-label. A step splitter segments each chain into claims. For math this is rule-based extraction of equations and derived quantities, falling back to model segmentation. For commonsense tasks it is model-extracted propositions. Each claim $c_j$ becomes a probe: the original question plus only the premises that claim depends on, phrased as a direct question ("Given … what is the total cost?" or "Is it true that …?"). The probe never includes later steps, the final answer, or the original wording of the claim. We sample $k$ answers to each probe in fresh contexts and define the claim's support $s_j$ as the fraction that agree with the claim. The chain's soundness score is its weakest link, $S = \min_j s_j$. The rationale for the minimum is that a single rationalized step is enough to make a chain a false positive.

$S$ is used in two ways. In the filtering variant, only chains that both match the majority answer and have $S \ge \tau$ enter supervised fine-tuning or DPO pairs, mirroring CORE-PO's pipeline. In the reinforcement-learning variant, the GRPO reward is answer agreement multiplied by $S$, mirroring TTRL and Co-rewarding. A cheaper gated version runs probes only on questions with low direct-answer entropy, where the confirmation-bias study predicts rationalization.

The first phase is diagnostic and needs no training. We use labeled sets but hide the labels from the method. The test is whether $S$ identifies wrong majorities better than the alternatives, measured separately for low-entropy and high-entropy questions. The second phase is iterative self-training, tracking pseudo-label precision against the hidden labels at every round. The third phase repeats the comparison at larger scale (32B to 70B). The entrenchment paper predicts that larger models execute flawed strategies more stably, which could either make blinding more valuable or make blind probes fail together with the chain.

Baselines

The naive reference is the untuned model, scored with both greedy decoding and self-consistency at matched inference samples. Large Language Models Cannot Self-Correct Reasoning Yet reports 88.2% on GSM8K for nine-sample self-consistency with GPT-3.5. We use it as a calibration of how strong plain voting is, not as a number to beat on our backbones.

The strongest vote-based competitor is TTRL-style majority-reward self-training as cataloged in the survey, which reports MATH-500 rising from 46.7% to 83.4%. We rerun it on our own backbones with the same unlabeled pool. Its token budget will match ours: the samples spent on probes are instead given to TTRL as extra chains for voting, so any gain we show cannot be explained by extra compute. Each method gets the same number of tuning trials over thresholds and learning rate.

The strongest step-level competitor is CORE-PO. On GSM8K it reports 93.5% with Qwen2.5-7B-Instruct and 90.5% with Llama3.1-8B-Instruct, and 55.8% on MATH with Qwen2.5-7B-Instruct, all under inference-time scaling. We use those backbones so the comparison is direct, and we reproduce CORE-PO rather than relying only on its reported numbers. The strongest stability-focused competitor is Co-rewarding, which reports a GSM8K Pass@1 of 94.01% on Qwen3-8B-Base. We run Co-rewarding-III on that backbone with our compute budget. On StrategyQA and CommonsenseQA, no reported result was found on Lacuna for any of these self-training methods, so all numbers there come from our reruns.

The key ablation isolates blinding. We take the same extracted claims and score each one with the model's in-context probability that it is true, with the full chain visible, then score it again with the probe. The two versions are identical except for what the model can see. A second ablation replaces the weakest-link minimum with the mean. A third replaces the probes with random claims of matched length, to rule out a generic regularizing effect.

Beating the baselines means three things. First, at matched tokens, pseudo-label precision is higher in later rounds than both TTRL and CORE-PO. Second, final accuracy matches or exceeds them on MATH500 and GSM8K. Third, it clearly exceeds them on low-entropy StrategyQA and CommonsenseQA questions. If blinding gives no edge over in-context scoring of the same claims, the core claim is wrong.

Why not just…?

Why not just use majority voting, or TTRL, which already works? Voting removes brittle errors and is a real threat on math, where the confirmation-bias study finds formal rules constrain rationalization. The survey's false-popular modes are exactly the cases voting cannot fix, but how common they are on GSM8K-level math is an empirical question. The diagnostic phase settles it. If $S$ adds no discrimination beyond vote margin on wrong majorities, the method is not worth its cost there, and we will say so and restrict the claim to prior-dominated tasks.

Why not just ask the model to rate its own steps, as CORE-PO does? That is the ablation above. The confirmation-bias study's finding that models discount rationales conflicting with their hunch predicts that in-context ratings will inflate support for rationalized steps. If the blinded and unblinded scores perform equally, the prediction fails, and CORE-PO already covers this ground.

Why not just have a second, different model act as peer reviewer, as in co-teaching (Co-teaching)? A stronger or different model is a legitimate competitor, and we run it. A cross-family judge of similar size scores the same chains under the same budget. There are two reasons to expect it may not dominate. Huang et al. find that multi-agent debate amounts to voting in disguise, because the agents share pretraining biases. A much stronger judge changes the problem from self-training to distillation. If a same-size peer matches blinding, the lesson is that independence matters more than blinding specifically, and the two can be combined.

Why not just prompt the model to critique and revise before training? Huang et al. report GSM8K accuracy falling from 95.5% to 89.0% for GPT-4 and from 62.0% to 36.5% for Llama-2-70b under intrinsic self-correction, and the entrenchment paper finds self-refinement amplifies coherent errors. Critiquing with the full chain in view is the failure case this proposal is designed around, so we include it only as a sanity baseline.

Proposed Experiments

Training pools are unlabeled questions from MATH, GSM8K, and StrategyQA. Evaluation uses GSM8K, MATH500, AMC, StrategyQA, CommonsenseQA, and ConsisEval. ConsisEval tests whether self-training improves agreement between hard problems and their easier subproblems; the original paper reports 7B models scoring 50–70% on its consistency score.

Metrics are final accuracy and pseudo-label precision per round, using hidden labels. We also measure the fraction of wrong majorities that $S$ catches, split by direct-answer entropy. To track sound accuracy, we audit intermediate logic in a code-assisted variant of the math runs with the code-structure judge from Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics. That paper found final accuracy hides unsound programs.

The first decisive check is the diagnostic AUROC of $S$ against vote margin and in-context truth probability on wrong-majority questions. Its predicted signature is a gap that is large at low entropy and on commonsense tasks, and small on high-entropy and math questions. The second is the precision curve over training rounds. Vote-based training should drift downward on prior-dominated questions while blinded filtering holds steady. The third is the 70B comparison. If blind probes start agreeing with rationalized chains at scale, that would show coherent errors at scale are rule-level rather than hunch-driven, and that blinding does not address the scale-reversal hazard.