Reasoning models now routinely "think" for thousands of tokens, and much of that thinking is revision: "Wait, let me check," followed by a second or third attempt. Two bodies of evidence point in opposite directions. One says revision mostly hurts. Models talk themselves out of correct answers more often than they fix wrong ones, and trimming the revision loops saves tokens at little cost in accuracy. The other says learned backtracking can raise accuracy sharply. This note proposes a way to teach a model, from its own outputs, which of its revisions actually help. The method borrows the central lesson of semi-supervised learning: labels a model generates for itself should stay provisional and be withdrawn when they stop holding up. Freezing them locks early mistakes into training.
Concretely, we cut each self-generated reasoning trace at its revision points. At every cut we ask the current model to answer immediately, then check that forced answer against a verified final answer. Each revision segment is labeled needed (it turns a wrong answer right), harmful (it turns a right answer wrong), or redundant (the answer was already right and stays right). These labels are recomputed from the current model at every training round. A segment whose label changes between rounds is withdrawn from the loss until it settles. We train and test mostly on freshly generated problems that cannot be in any pretraining corpus. The reason is that we suspect, and will test first, that the familiar "revision hurts" pattern is partly inflated by memorized benchmark answers.
Core Claim
The claim has two parts. The first is a measurement. On static, likely-contaminated benchmarks, a model's first answer is often recalled rather than derived, so any revision can only damage it. That makes the correct-to-incorrect flip rate reported by Large Language Models Cannot Self-Correct Reasoning Yet partly an artifact of contamination. On structurally matched fresh problems, the ratio of helpful to harmful revisions should shift toward helpful. If this holds, stopping policies tuned on static math sets are miscalibrated for novel problems, because they learned "never revise" where revision sometimes helps.
The second part is a method. Per-segment revision labels probed from the current model and recomputed each round with demotion of unstable labels should give a better trade-off than two existing approaches. The first is static truncation labels computed once offline. The second is outcome-level reinforcement learning, which cannot tell a helpful revision from a harmful one inside a trace that happens to end correctly. "Better" means fewer correct-to-incorrect flips, preserved incorrect-to-correct fixes, and fewer tokens.
Where This Sits
Three papers motivate the gap. Large Language Models Cannot Self-Correct Reasoning Yet shows that prompting-based self-correction lowers accuracy without an oracle, for example GPT-4 on GSM8K falling from 95.5% to 89.0%. It uses static datasets and does not control for contamination. GPT-4 Doesn't Know It's Wrong shows the same failure on graph coloring, where GPT-4 verified only 2 of 100 correct colorings. DyVal generates reasoning problems from random computation graphs and finds that models scoring well on GSM8K score near zero on fresh instances. None of these asks whether self-correction failure changes character between memorized and fresh items. We ask exactly that.
Three overthinking methods constrain and compete with the proposal. Self-Braking Tuning observes that the first complete solution is usually right. It masks later revision segments using stop points computed once, offline, from heuristic marker ratios. Our method differs in two ways: its labels are grounded in answer probes rather than marker counts, and they are refreshed as the model changes. Cumulative Entropy Regulation (CER) uses on-policy reinforcement learning. It penalizes high uncertainty late in correct responses, but its reward covers the whole trajectory, so it cannot assign credit to individual segments. Stop Spinning Wheels (RCPD) decides at inference time when to exit, using the rank of the end-of-thinking token. It learns nothing about which revisions to keep.
Pulling the other way, Self-Backtracking trains models to revert to earlier states and gains over 40% on Countdown. That paper is direct evidence that some revision is valuable. We take it seriously, and our "needed" label exists to preserve that value.
The method comes from semi-supervised learning. Meta Pseudo Labels shows that labels a teacher assigns must adapt to how the student is actually doing, or early errors get reinforced. Uncertainty-aware pseudo-label selection returns an example to the unlabeled pool when uncertainty about it rises again. We reuse the demotion idea: instability across rounds is the uncertainty signal. We do not reuse Meta Pseudo Labels' costly two-level optimization.
Rationales for Answers to Simple Math Word Problems Confuse LLMs supplies a warning. Models trained only on correct solutions become "yes-men" that accept flawed reasoning. Our "harmful" label therefore gives the model explicit examples of revisions that should not have happened.
Research Program
Stage one: the contamination diagnostic. We build matched pairs of problems. Each pair contains a GSM8K test item and a structural isomorph drawn from DyVal's arithmetic and linear-equation generators at matched graph depth. To check that the generator itself carries no exploitable artifacts, we follow the no-candidate and paired-scoring logic of Back to Square One. We also check that models do not show near-zero surprise on the fresh items, a memorization signal used by Yax et al.. We then take native thinking traces from Qwen3-4B, Qwen3-8B and Llama-3.1-8B-Instruct and apply the probing procedure described next. From this we measure, per item, how often revisions fix a wrong answer versus break a right one, on static versus fresh items.
Stage two: probe-labeled segments. Each trace is split at revision markers into segments $s_1,\dots,s_K$. After each prefix ending at $s_k$, we force a short answer with greedy decoding and record $c_k \in \{0,1\}$, whether that answer matches the verified answer. Segment $k$ is labeled needed if $c_{k-1}=0$ and $c_k=1$, harmful if $c_{k-1}=1$ and $c_k=0$, and redundant if both are 1. The training target is built as follows. Keep all segments through the last needed one. Replace any harmful segment and everything after it with a stop at the preceding point. Mask redundant segments that come after the stop. This costs $K$ short forced answers per trace, typically fewer than ten, which is far cheaper than the second-order gradients of Meta Pseudo Labels.
Stage three: refresh and demotion. In each round $r$, the current model resamples traces for the training problems and relabels them. A problem's segment labels count as stable if the needed/harmful/redundant pattern at aligned segment positions agrees across rounds $r-1$ and $r$. Unstable problems are removed from the loss for that round and remain eligible later. Training data mixes the GSM8K and MATH training splits with a large, continuously regenerated pool of DyVal-style problems. The only external signal we assume is a verified final answer, the same assumption CER makes.
Baselines
The naive reference point is the base model's native thinking, with no intervention. Next to it we run direct answering with thinking disabled, and a "stop at first complete solution" rule. Together these bracket the space: always revise, never think, and never revise.
The strongest compute-heavy competitor is Self-Consistency, which samples several answers in parallel and takes a majority vote. Large Language Models Cannot Self-Correct Reasoning Yet reports 88.2% on GSM8K with nine votes, against 83.0% for multi-agent debate at the same budget. We run it at the same total generated-token budget as our method. We count beating it as matching its accuracy at a fraction of the tokens, or exceeding it at equal tokens.
The strongest training-based competitor is CER from Explore Briefly, Then Decide. It reports Qwen3-4B on GSM8K rising from 92.80% to 94.09% while cutting response length by up to 71%. We retrain it on exactly our data mixture, with the same number of sampled traces per problem and the same hyperparameter search budget. Beating it means lower correct-to-incorrect flip rates at no loss in accuracy or length on fresh problems. On static GSM8K, merely matching it is acceptable.
Self-Braking Tuning is the closest method. It reports a 62.8% token reduction on Llama-3.1-8B-Instruct while retaining 94.1% of baseline accuracy. We reimplement it on our data and backbone. RCPD is an inference-time exit rule that reports about 35% token savings at full accuracy on AIME24 with Qwen3-8B. We apply it on top of both the base model and our trained model. We found no reported result on Lacuna for CER, SBT or RCPD on DyVal or any other contamination-free generated suite, so all fresh-data numbers will be new.
Two ablations isolate the key ingredients. The first freezes labels: we compute probe labels once, from the round-zero model, and never refresh or demote them. This is the direct test of whether dynamics matter. The second replaces probe labels with SBT's marker-ratio heuristic inside our refresh loop. This tests whether grounding labels in answer probes matters.
Why not just…?
Why not just skip thinking and answer directly? On some tasks this wins outright. Mind Your Step (by Step) reports a 36.3% absolute drop for o1-preview relative to zero-shot GPT-4o on implicit statistical learning. But OptimalThinkingBench shows that non-thinking models hit a ceiling on hard problems. Our deeper DyVal tiers and the UnderthinkingBench split will show whether direct answering collapses where our method does not.
Why not just use Self-Consistency? It reliably beats iterative correction at matched compute, and it is a real threat. Our answer is efficiency, not accuracy: it multiplies cost by the number of samples. If it dominates our method at equal tokens on fresh problems, the method has failed.
Why not just fine-tune on fresh generated problems with ordinary answer-filtered traces? DyVal already shows that such fine-tuning transfers. If contamination is the whole story, this simple fix should capture most of the gain. We run it as a baseline on the identical fresh pool. The gap between it and our method measures what segment labeling adds beyond clean data.
Why not just use on-policy reinforcement learning like CER, which is already "dynamic"? It does adapt to the current model, which weakens our novelty argument. What it lacks is credit inside a trajectory. A correct trace containing one harmful and one needed revision receives a single reward. The segment-level flip-rate comparison below settles whether that difference matters in practice.
Decisive Checks
We evaluate on four sets. The first is DyVal tiers D1 to D4 across arithmetic, linear equations, Boolean logic and reachability, freshly regenerated at test time. The second is static GSM8K and MATH500. The third is AIME24/25, used only for underthinking checks. The fourth is OptimalThinkingBench, scored with its $F_1^{otb}$ metric, where o3 reports 71.1. Transfer to a different structure is tested on the graph coloring set of Stechly et al., with external checking used only for scoring. On every set we report accuracy, generated tokens, and the two flip rates.
Three outcomes decide the proposal. The first concerns stage one. If the ratio of helpful to harmful revisions does not differ between static items and their fresh isomorphs, the contamination half of the claim is false, and the paper becomes a pure method study. The second is the frozen-label ablation. If it matches the full method on fresh-data flip rates, the pseudo-label refinement ingredient is unnecessary, and we will say so. The third is the comparison with CER, retrained on the same fresh pool. The method earns its place only if it lowers correct-to-incorrect flips on DyVal D3 and D4 without reducing incorrect-to-correct fixes.
References
- Large Language Models Cannot Self-Correct Reasoning Yet
- GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Let LRMs Break Free from Overthinking via Self-Braking Tuning
- Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulation
- Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit
- Step Back to Leap Forward: Self-Backtracking for Boosting Reasoning of Language Models
- Meta Pseudo Labels
- Positive-Unlabeled Learning with Uncertainty-aware Pseudo-label Selection
- Rationales for Answers to Simple Math Word Problems Confuse Large Language Models
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- Studying and improving reasoning in humans and machines
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worse
- OptimalThinkingBench: Evaluating Over and Underthinking in LLMs