When a language model works through a multi-step problem, one wrong step early in its written reasoning often commits it to a doomed path. By the time it writes the final answer, that answer can still carry a high probability. Current confidence estimators mostly read the model once, at the end. They can say that a trace is probably wrong, but not where it went wrong, so they offer little to act on beyond throwing the whole trace away. Classical parsing research faced a similar problem and found a fix. Punctuation marks act as boundaries that word-to-word relations rarely cross, and getting the boundary decisions right removes most downstream errors. In one study, nearly the entire accuracy gain from enforcing correct punctuation came from errors that cascaded from the boundary, rather than from the boundary arcs themselves.
This note proposes treating the clause and punctuation boundaries inside a model's reasoning trace as checkpoints. At each one, a small trained classifier reads the model's hidden state and estimates the probability that the reasoning written so far can still reach a correct answer. If that estimate collapses, generation backs up to the last boundary where the estimate was high and samples a new continuation from there. The project tests two claims. The first is that boundary-located confidence estimates are well calibrated, meaning stated probabilities match observed success rates. The second is that backing up to boundaries repairs reasoning more cheaply than resampling whole traces. The evaluation also borrows parsing's split between direct and cascaded gains to measure how much of the repair comes from stopping a snowball.
Core Claim
Calibration research has shown that a frozen model's hidden states contain a good correctness signal. Uncertainty Calibration for Tool-Using Language Agents trains a 3-layer probe on hidden states from about 500 examples and reduces Expected Calibration Error (ECE: the average gap between stated confidence and actual accuracy) on TabMWP from 0.690 to 0.035. That probe reads the model only at the end of a prompt or trace. Separately, parsing and attention work shows that punctuation tokens are where structure gets resolved. Identifying Cascading Errors using Constraints in Dependency Parsing finds that cascaded gains account for 1.5 of a 1.8-point improvement in Unlabeled Attachment Score (UAS, the fraction of words attached to the correct head) when punctuation is enforced. Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models shows that pooling punctuation tokens separately preserves information that block averaging blurs.
Our claim is that a correctness probe trained on boundary-token states gives calibrated estimates mid-trace, and that these estimates support two uses. First, they locate the first fragment where success probability drops. Second, backing up to the preceding boundary recovers accuracy at lower compute than whole-trace resampling, with most of the gain appearing downstream of the repaired fragment, in the same way cascaded gains dominated in parsing.
Prior Work and How This Differs
Two papers already calibrate at step granularity, and they are the closest work. MMBoundary calibrates vision-language models sentence by sentence. It uses entropy, log-probability and image-text similarity signals, and trains with SFT and PPO, reporting a 7.5% reduction in multi-granularity ECE. LoGU splits long answers into atomic claims using GPT-4o and trains the model to express uncertainty on the wrong ones. These two papers motivate the step-level view. We differ from them in three ways. Our setting is text-only reasoning with no visual grounding. Our boundaries come from the text's own punctuation and clause structure, not from an external model. Our confidence estimate is used to decide where generation backs up, not to train how the model verbalizes uncertainty.
Fact-and-Reflection imposes prompt-level stages and reduces ECE by 23.5%. Its boundaries are macro workflow stages, whereas ours fall inside the generated text.
Three results constrain the design. Quantifying Faithful Confidence Expression in Large Reasoning Models reports that prompting interventions give near-zero improvement in faithfulness for reasoning models. It also finds that representation probes, log-probabilities and prefix-conditioned sampling disagree with each other. We therefore avoid verbal confidence entirely. We train the probe on prefix-conditioned sampling outcomes, the most behavioral of the three estimators. Large Language Models Must Be Taught to Know What They Don't Know shows that verbal confidence on open-ended MMLU sits near 30% ECE, and that roughly 1,000 graded examples suffice for a transferable estimator. This supports a small-label probe. Evaluating the role of non-lexical markers in GPT-2's language modeling behavior finds that removing punctuation flattens entropy after about 10 words of context. This suggests boundaries matter for how the model's uncertainty evolves. It also warns that traces with sparse punctuation may weaken the method, which we test directly.
Research Program
The target for each boundary must not depend on human step annotations. For a trace prefix ending at boundary $b$, we define the prefix value
$$v(b) = \frac{1}{K}\sum_{k=1}^{K} \mathbb{1}[\text{completion } k \text{ from prefix } b \text{ reaches the correct answer}],$$
where $K$ is the number of sampled continuations (we plan $K=8$). $v(b)$ is an empirical success probability, so a calibrated probe should predict it directly.
Boundaries are found by a rule-based segmenter. It cuts at sentence-final punctuation, semicolons and colons, and at commas that a lightweight parser marks as clause-level. This follows the inter-punctuation fragment notion and the "loose" constraint from Punctuation: Making a Point in Unsupervised Dependency Parsing. Under that constraint, each fragment has a single internal head that may attach outside the fragment. Mathematical expressions are protected from splitting.
The probe follows the recipe from Uncertainty Calibration for Tool-Using Language Agents: a 3-layer MLP on frozen middle-to-late layer states, trained on a few hundred problems. We read the state at the boundary token itself and also concatenate the mean of the fragment's token states. The probe is trained with a Brier-style loss against $v(b)$.
The repair procedure works as follows. Generation proceeds normally. At each boundary, if the probe's estimate falls by more than a threshold relative to the last high-confidence boundary, we roll back to that boundary and resample the next fragment, up to a fixed token budget.
The analysis borrows the parsing decomposition. Repair gains are split into direct gains, meaning the rewritten fragment is now correct, and cascaded gains, meaning later fragments become correct without being rewritten. Fragment correctness is judged by whether $v$ recovers.
Baselines
The naive reference point is the final-answer token probability under plain chain-of-thought. It is computed on the same traces and calibrated with temperature scaling on a held-out split. This is the "do nothing" estimator that the compounding-logit critique says should fail.
The strongest existing estimator is PROBECAL's end-of-trace probe from Uncertainty Calibration for Tool-Using Language Agents, which reports a TabMWP ECE of 0.035 and an accuracy gain of over 7% against TroVE. On MATH, no separate reported result is found on Lacuna for PROBECAL. It will be retrained on our models, Llama-3-8B and Mistral-7B as in that paper, using the same problems, the same probe size and the same tuning budget. For mid-trace comparison it will also be applied at every boundary, using a model trained only on final states.
The strongest repair competitor is compute-matched best-of-N with self-consistency voting, and separately with the end-of-trace probe used as a reranker. Both get exactly the same total generated-token budget as boundary rollback.
For StrategyQA we also run Fact-and-Reflection using its released code (fact-and-reflection). Its reported reference is a 23.5% relative ECE reduction over standard prompting across StrategyQA and Web Questions.
Two ablations isolate the key ingredient. The first is a matched-density control: probes at randomly chosen token positions, with the same number of positions per trace. The second is a newline-only segmentation in the style of MMBoundary. Together they test whether the gains come from the boundary positions themselves or simply from reading the model densely.
Beating the baselines means three things. Mid-trace ECE and Brier score must be lower than the end-of-trace probe applied mid-trace, and AUROC for predicting eventual failure from early prefixes must be higher. Repair must deliver higher accuracy than best-of-N at equal tokens. The boundary probe must outperform the random-position control by more than seed variance.
Why not just…?
Why not just use PROBECAL at the end of the trace? It is already well calibrated for whole traces, and for pure accept/abstain decisions it may be enough. We will report that case honestly. What it cannot provide is a location, and therefore a cheap repair. The decisive test is the compute-matched repair comparison. If reranking whole traces with the end-of-trace probe matches boundary rollback in accuracy per token, the repair claim fails, and the contribution reduces to mid-trace failure prediction.
Why not just probe every token, or random positions? This is the most serious threat. A probe at arbitrary positions might be as calibrated as one at boundaries, because hidden states change smoothly between tokens. The matched-density random-position ablation settles it. If random positions match boundaries, the claim narrows: boundaries then matter only as natural, readable rollback units. The rollback experiment will then compare backing up to boundaries against backing up a fixed number of tokens.
Why not just ask the model, or use token entropy? Verbal confidence sits near 30% ECE on open-ended MMLU (Large Language Models Must Be Taught to Know What They Don't Know). Hedging prompts give near-zero faithfulness improvement in reasoning models (Quantifying Faithful Confidence Expression in Large Reasoning Models). Per-fragment mean token entropy is cheap, however, so it is included as an estimator at the same boundaries and must be beaten.
Proposed Experiments
The first experiment tests calibration. On GSM8K, MATH and TabMWP, we measure ECE, Brier score and reliability diagrams of each estimator against $v(b)$, reported separately for the first, middle and last thirds of traces. To our knowledge, no reported GSM8K calibration result is found on Lacuna. AUROC for eventual failure, computed from the first boundary at which the estimator drops, measures how early it localizes errors.
The second experiment tests repair. We report accuracy against generated tokens for boundary rollback, best-of-N and self-consistency, together with the direct and cascaded decomposition. The prediction borrowed from parsing is that cascaded gains will dominate. A result where most gains are direct would mean that rollback just fixes local slips, and the snowball story would not hold.
The third experiment is a robustness check motivated by the uncertainty-plateau finding. We strip punctuation from traces before segmentation and probing. The aim is to see whether calibration degrades, which would show the method depends on surface punctuation, or whether the probe recovers clause boundaries from the model's internal states anyway.
StrategyQA, compared against Fact-and-Reflection, tests transfer from math to implicit multi-hop reasoning.
References
- Uncertainty Calibration for Tool-Using Language Agents
- Identifying Cascading Errors using Constraints in Dependency Parsing
- Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models
- MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration
- LoGU: Long-form Generation with Uncertainty Expressions
- Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Models
- Quantifying Faithful Confidence Expression in Large Reasoning Models
- Large Language Models Must Be Taught to Know What They Don’t Know
- Evaluating the role of non-lexical markers in GPT-2's language modeling behavior
- Punctuation: Making a Point in Unsupervised Dependency Parsing