Finding the First Wrong Step by Where Different Models Disagree

When a language model solves a multi-step problem, one early slip usually ruins the rest of the solution, because each later step is written to stay consistent with what came before. Models are surprisingly good at repairing such a slip once someone points to it. Given the exact index of the first error, accuracy on previously failed traces rises by 43.92 percentage points on one task. They are poor at finding the slip themselves: GPT-4 locates the first error only 52.87% of the time (LLMs cannot find reasoning errors, but can correct them given the error location). This proposal replaces the oracle with something cheap and automatic. We run several small models from different families on the same problem in parallel, line up the intermediate facts each one derives, and take the earliest step that most of the other models fail to reproduce as the predicted error location. We then repair the solution only from that step onward, and only on problems where such a disagreement exists.

The idea is borrowed from speech recognition. There, "system combination" merges the outputs of several recognizers by aligning their word hypotheses and voting position by position. This is ROVER, Recognizer Output Voting Error Reduction, and its relative confusion network combination (Kilgour et al.; Dufour et al.). Large language model (LLM) reasoning ensembles almost always vote only on the final answer. We propose voting on intermediate steps. The purpose of that vote is mainly to localize errors rather than to choose an answer, and the localization then drives a single targeted backtrack. If this works, it closes the gap between error detection and error correction without a trained reward model. It also keeps the extra latency of refinement confined to the minority of problems that need it.

Core Claim

For problems whose intermediate steps produce checkable facts, the first step whose derived fact is unsupported by most of a heterogeneous model pool is a far better guess at the true first error than self-critique. We expect it to beat GPT-4's 52.87% localization on BIG-Bench Mistake. Backtracking from that step gives most of the oracle-backtracking gain at a fraction of the token and wall-clock cost of whole-chain refinement or large-N self-consistency.

We define the support score as follows. Let system $i$ produce a chain whose step $k$ yields an intermediate fact $f_{ik}$: a computed quantity, an object-location state, or a partial sorted list. Let $F_j$ be the set of facts derived anywhere in system $j$'s chain. The support score is

$$ s(f_{ik}) = \frac{1}{K-1}\sum_{j \neq i} \mathbb{1}\left[f_{ik} \in F_j\right], $$

where $K$ is the number of systems. The predicted first error in chain $i$ is the smallest $k$ with $s(f_{ik}) < \tau$, for a threshold $\tau$ tuned on held-out data.

Matching against the set $F_j$ rather than against position $k$ is the point. Classical ROVER alignment assumes outputs proceed in the same order, and reasoning chains do not: one model may compute a subtotal at step 2 and another at step 4. Fact-set matching avoids that failure while keeping the voting logic.

Relation to Prior Work

The closest work is MAgICoRe. It sorts problems into easy and hard using outcome and process reward models (separately trained scorers of whole answers and of individual steps), resolves easy problems by weighted voting, and sends hard ones through a solver–reviewer–refiner loop. It shows that selective refinement matters: universal refinement lost 5.2% on MATH. It also shows that one round of selective refinement beats 120-way self-consistency by 3.4% on Llama-3-8B at about half the compute.

We keep the selectivity but change where both the difficulty signal and the error location come from. Both come from cross-model disagreement on intermediate facts rather than from trained reward models, and the repair is a backtrack from one step rather than an agent rewrite. MAgICoRe therefore motivates the design and is the strongest competitor.

Two other lines of work supply method pieces. Tyen et al. supply the backtracking procedure: regenerate the flagged step at high temperature, then continue at low temperature. We reuse it unchanged so that any difference comes from localization alone. Deductive Verification of Chain-of-Thought Reasoning shows that checking steps in isolation lifts verification accuracy from 52% to 69% overall, and to 84% on GSM8K. Its Unanimity-Plurality Voting, however, makes an LLM verification call for every step of every chain, which is the latency cost we try to avoid by checking steps against other models' derived facts instead of against an LLM judge.

GRACE scores candidate next steps with a trained discriminator and gains 5.4% on GSM8K for LLaMA-7B. It samples and scores candidates at every step, and it cannot recover if no sampled candidate is correct. We intervene at one step per problem, and we draw alternatives from other model families.

SIRIUS is the closest in spirit on latency. A fast sparse model drafts, a dense model checks token likelihoods and rolls back, and correcting 11% of tokens recovers the dense model's accuracy (sparse Llama-3-8B rises from 38% to 72% on GSM8K). It operates at the token level, within one architecture, and needs white-box access to the key-value cache. Our method works at the step level, across model families, and needs only text outputs.

Three findings constrain the design. First-Step Advantage reports that Mistral-7B gets about 40% of GSM8K right on its first attempt but nearly 90% across 35 samples. The correct step therefore usually exists somewhere in a pool, which is what makes pool-based localization plausible. The diminishing-returns literature, including Deliberation Networks and Tang, is why we cap repair at one or two rounds. The speech literature warns that correlated systems vote errors in: Dufour et al. report an oracle combination gain of 12.6% against an achieved 1.7% relative ROVER gain. We read that gap as evidence that voting on outputs wastes most of an ensemble's information, and that pointing to where systems differ may recover more of it.

Research Program

We build a pool of $K$ open 7–8B models from different families. The file names Llama-3-8B, Mistral-7B, Qwen2-7B and OlMo-7B. Each model is prompted to write one step per line and to end each step with its derived fact in a fixed format. For arithmetic tasks the fact is a number with its unit. For BIG-Bench Mistake's tracking, sorting and Dyck tasks it is the current state, which can be compared exactly. Facts are matched by exact normalized equality, so no LLM judge is involved.

The pipeline is inexpensive. All $K$ chains are generated in parallel, so wall-clock time is bounded by the slowest drafter, not by $K$. A problem counts as easy and stops immediately if the final answers agree and every step of the leading chain clears $\tau$. Otherwise we take the plurality chain, find its first low-support step, and backtrack from there. The verified prefix is reused through prefix caching where the serving stack allows it. We test two repair variants: Tyen-style resampling, and substituting the competing fact with the highest support as a hint. We allow at most two rounds.

The analysis has three parts. First, localization accuracy as a function of $K$, of family diversity, and of $\tau$. Second, how often errors are shared across families at step 1. This is the correlated-error rate, which bounds the method. Third, a cost accounting of tokens, wall-clock latency and number of model calls per solved problem.

Baselines

The naive reference is greedy chain-of-thought from the best single model in the pool, with no combination and no refinement. Every method must beat it, and it fixes the cost floor.

The strongest existing method is MAgICoRe (Chen et al.). Its reported numbers are the bar: on Llama-3-8B, one iteration beats 120-way self-consistency by 3.4% and 120-way Best-of-k by 3.2%, and it gains 6.4% over the strongest baseline on MATH. We will rerun it with Llama-3-8B as solver on GSM8K, MATH and SVAMP, using its own reward models, and report accuracy against both tokens and wall-clock time. Beating it means equal or higher accuracy at lower total tokens and lower median latency, or higher accuracy at matched tokens. Matching it without any trained reward model would already be a meaningful result.

Self-consistency is run at a token budget matched to our method, both on a single model and pooled across families. The pooled version is the key ablation. Heterogeneous terminal voting uses exactly the same $K$ chains as our method but votes only on final answers. The difference between the two isolates the contribution of step-level localization plus backtracking.

For localization specifically, two references apply. One is GPT-4 self-localization on BIG-Bench Mistake, at the reported 52.87% (Tyen et al.). The other is a trained step critic. For the critic, the file reports only qualitative transfer for the small PaLM 2 Otter, so we will instead train a step classifier on the BIG-Bench Mistake training split under the same compute budget. Oracle-location backtracking is the upper bound. On Tracking Shuffled Objects it gains 43.92 points on incorrect traces, and we report what fraction of that gain we recover. Deductive Verification's Unanimity-Plurality Voting is the step-level baseline. The file gives 84% step-verification accuracy on GSM8K but no final-accuracy number for this setting, so we will run it ourselves at matched sample count and report its token multiplier.

Why not just…?

Why not just pool the models and majority-vote the final answers? This is the most serious threat, because heterogeneity alone may produce most of the gain. Our design measures it directly: the pooled terminal vote uses the identical $K$ chains. If localization plus backtracking adds less than about one point over it on GSM8K and MATH, we will report that step-level fusion buys nothing beyond diversity. We would then reframe the contribution around localization on BIG-Bench Mistake, which terminal voting cannot do at all. The evidence that it should add something: terminal votes can reward chains whose two errors cancel (Ling et al.), and they cannot help when the majority is wrong, whereas a single fixed step can.

Why not just sample one model many times and look for where its samples diverge? This removes the cost of serving several models and is a real competitor. Its weakness is correlated errors. Samples from one model share the same high-probability wrong steps, the failure GRACE documents (Khalifa et al.), and that is exactly the case where agreement points to the wrong place. We run the same localizer on $K$ samples from Llama-3-8B and compare it with $K$ cross-family chains, alongside the measured step-1 error correlation. If single-model divergence localizes as well, we will adopt it as the cheaper method.

Why not just train a process reward model or step critic, as MAgICoRe and GRACE do? Trained critics can work well, but they need labelled step data for each domain, and running them adds a sequential model call. Our comparison gives the trained critic the BIG-Bench Mistake training data, so it receives in-domain supervision our method never sees. The comparison also covers transfer to held-out BIG-Bench Mistake tasks, where Tyen et al. suggest critics transfer only partly.

Why not just refine every answer a few times with self-critique? The file already argues against it. Universal refinement cost 5.2% on MATH (MAgICoRe), self-localization is near chance, and gains plateau after one or two passes (Xia et al.). We include two-round self-refine anyway for completeness.

Proposed Experiments

The decisive first experiment is on BIG-Bench Mistake, which has 2,186 annotated traces. For each trace, we run the $K$ pool models on the same question and treat the annotated trace as chain $i$. We then predict its first error as the first step with support below $\tau$. Success means clearly exceeding 52.87% localization accuracy using only 7–8B models. We also feed the predicted locations into Tyen backtracking and report the recovered share of the 43.92-point oracle gain on Tracking Shuffled Objects. If localization does not beat GPT-4 self-critique here, the project stops.

The second experiment is end-to-end on GSM8K, MATH and SVAMP against all baselines. It reports accuracy, total tokens, median and 95th-percentile wall-clock latency, and the fraction of problems routed to repair. We expect that fraction to be small on GSM8K and larger on MATH. A falsification check is that accuracy on problems routed as easy must not fall below greedy accuracy, which tests the over-correction concern. To probe the late-error regime, where Jin et al. and Tyen et al. expect step-1 fixes to fail, we stratify results by chain length. We also report localization accuracy separately for errors after step 4.

The last experiment varies $K$ from 2 to 5 and swaps families for same-family checkpoints. It traces how localization accuracy depends on error correlation, which is the quantity the speech-recognition literature says determines whether combination helps at all.