Language models solve multi-step problems better when they write out their reasoning first, a technique called chain-of-thought (CoT) prompting. The standard way to make this more reliable is self-consistency: sample many full reasoning chains and take a majority vote on the final answers. That vote happens only at the end. It discards every intermediate step, and it assumes errors scatter randomly across samples. On logic problems they often do not. When a model is drawn toward a wrong but locally valid branch, most samples walk down the same dead end, and the vote certifies the mistake.
We propose to move the vote from the final answer to each reasoning step, and to draw the voters from different subsets of the few-shot examples rather than from repeated samples of one prompt. The pool of worked examples is split into small groups ("buckets"). Each bucket, placed in front of the same partial solution, acts as a separate expert. At every step the buckets propose next steps and score each other's proposals. The chain continues with the step they jointly find most plausible, and extra compute is spent only at steps where the buckets disagree. The bet is that part of the pull toward dead ends comes from the specific examples in the prompt, and that experts conditioned on different examples will disagree exactly at the branch points where one prompt alone commits greedily.
Core Claim
Two ingredients carry the proposal. The first is where aggregation happens: at step boundaries during decoding, not over finished answers. The second is where diversity comes from: disjoint demonstration buckets, not temperature sampling. Our claim is that combining the two gives fewer branch-choice errors on multi-hop deduction, at equal or lower compute, than full-chain voting over the same buckets or step-level search within a single prompt. The claim can be falsified directly. If bucket experts make the same wrong branch choice as often as repeated samples from one prompt do, the mechanism has nothing to exploit, and we will report that as the main finding.
Where This Sits
The evidence motivating the work comes from three analyses. Language Models Are Greedy Reasoners parses generated rationales on PrOntoQA into formal proofs. It finds that 93.2% of individual steps are valid on 5-hop fictional proofs, yet whole proofs fail because the model picks wrong branches. Sampling 40 paths with self-consistency did not repair this. Two Failures of Self-Consistency shows models stay consistent under substitution of intermediate results less than half the time. Together these say that errors live at particular steps and are correlated across samples. Terminal voting is badly placed to catch both.
Three methods aggregate reasoning chains, and all three vote on terminal answers. Self-Consistency samples many chains from one prompt and votes on answers. Early-stopping Self-Consistency stops sampling once a window of answers agrees, which inherits the failure of halting on a consistent wrong answer. Prefix Consistency weights each chain by how often regenerating its tail reproduces its answer, which inspects prefixes but still scores whole chains. We keep their shared insight that agreement signals reliability. We apply it to individual steps, before the chain commits.
Two methods supply our source of diversity. Exploring Demonstration Ensembling for In-context Learning (DENSE) splits demonstrations into buckets, computes an answer distribution per bucket, and combines them with a product or mixture of experts. It reports a 2.4-point average gain over plain concatenation on GPT-J, but only for single-shot answers without rationales. Boosted Prompt Ensembles shows that different CoT example sets cover different problems. We borrow DENSE's buckets and its product-of-experts rule, and run them per reasoning step instead of per answer.
An In-Context Learning Theoretic Analysis of Chain-of-Thought constrains the design. It bounds chain error by the single hardest step, which argues for concentrating compute where it matters rather than spreading it uniformly. The Curse of CoT finds that long CoT prompts hurt pattern-induction tasks by a relative 20.42%, partly through the distance between the query and the examples. Short buckets reduce that distance, so we test pattern-induction tasks as a secondary setting where the effect could go either way.
Research Program
Given a pool of worked CoT examples, we partition them into $b$ buckets $D_1,\dots,D_b$. The partition is either random or clustered by problem embedding, following DENSE. Each bucket's example prefix is encoded once and cached. Decoding proceeds step by step, where a step is one line of the rationale.
At each step every bucket greedily proposes a continuation of the shared partial chain $r$. If all proposals match after normalization (for PrOntoQA, identical parsed logical steps; for math, the same equation), the step is committed at the cost of $b$ short greedy continuations. If they differ, every candidate step $c$ is scored by all buckets:
$$s(c) = \frac{1}{b}\sum_{i=1}^{b} \frac{1}{|c|}\log p(c \mid D_i, x, r),$$
where $x$ is the question and $|c|$ is the token length of $c$. The highest-scoring candidate is appended. This is a length-normalized product of experts. A step survives only if no bucket finds it implausible, which is the property we want against example-specific greedy pulls.
A second variant samples $k$ candidates per bucket at contested steps, to test whether more candidates help. A third uses mixture (mean-probability) scoring, to test whether veto power matters.
We also build the diagnostic that decides whether the method has anything to work with. On PrOntoQA we locate the first branch point in each problem, meaning the first step where more than one valid deduction exists. We then measure how often wrong choices at that point are shared, in two cases: across buckets, and across temperature samples from one full prompt. If bucket errors are no less correlated than sample errors, step-level ensembling across buckets cannot beat self-consistency, and we stop there.
Baselines
The naive reference is greedy CoT with all examples concatenated into one prompt. Every method must beat it on accuracy to be worth any extra compute.
The strongest existing competitor is self-consistency over the concatenated prompt. It is simple, training-free, and the default that practitioners actually use. On the file's closed models, Self-Consistency reports 74.4% GSM8K accuracy on PaLM-540B. Boosted Prompt Ensembles reports 81.0% for self-consistency on code-davinci-002 and 85.2% for boosting, with 57.0% and 63.4% on AQuA. We work on open-weight models because we need token log-probabilities: Llama-2 7B, Qwen2.5-7B, and LLaMA-3.1-8B-Instruct. There is no reported result on Lacuna for self-consistency on these models on GSM8K, AQuA, or PrOntoQA, so we rerun every baseline ourselves. We use the same example pool and the same decoding length cap for all methods. Compute is measured as total forward-pass tokens, with prefill counted once when cached, and we sweep the number of samples so each baseline gets a full accuracy-versus-compute curve rather than a single point.
Early-stopping self-consistency and prefix-consistency weighting are the strongest efficiency-oriented competitors. We run both at their published window and truncation settings on the same models.
The key ablation separates our two ingredients:
- Bucket ensembling with terminal voting. Each bucket generates full chains and the answers are pooled by vote or by DENSE's weighted max. This keeps the bucket diversity and removes the step-level aggregation.
- Step-level scoring within one prompt. Candidate steps are sampled from the concatenated prompt and scored by that prompt alone, which is a step-wise beam over a single context. This keeps step-level aggregation and removes the bucket diversity.
Beating the proposal's rivals means higher accuracy than all of these at matched compute on at least one of the two main settings. Beating self-consistency specifically means improving PrOntoQA 5-hop strict proof accuracy, the setting where self-consistency is documented to fail. Winning only on GSM8K, where self-consistency already works well, would not support the central claim.
Why not just…?
Why not just run self-consistency with more samples? On multi-hop deduction, Language Models Are Greedy Reasoners found that 40 samples did not fix branch selection, because samples inherit the same greedy preference. More samples sharpen a biased vote. Our compute-matched curves test this directly. If self-consistency's curve rises to meet ours on PrOntoQA as samples increase, the objection holds.
Why not just vote over full chains from different buckets? This is the most serious threat. It is cheap, it is parallel, and it already uses the diversity we rely on. The claim for step-level voting is that it catches a wrong branch before the chain commits to it, whereas terminal voting can only choose among chains that have already diverged. That matters only if the correct branch is a minority in most buckets' own greedy chains but still earns high scores from the other buckets. The first ablation settles this. If terminal bucket voting matches step-level voting, the honest conclusion is that bucket diversity was the whole effect, and we will report it that way.
Why not just search over steps within one prompt? Step-level search from a single context already explores alternatives at branch points. But its scores come from the same context that produced the greedy bias, so a dead-end step keeps its high likelihood. The second ablation measures this, and the branch-point correlation diagnostic predicts the outcome in advance.
Why not just fine-tune, or inject a learned reasoning vector? CoT Vectors reports gains of about 4.6 points from a roughly 4K-parameter learned vector on Qwen2.5-Math-7B. Reasoning Vectors reports gains from adding the weight difference of an RL-trained model. Both need training or a donor checkpoint from the same model lineage. Both also change the model's default behavior rather than correcting individual branch choices. They are complementary to our method, not substitutes. We run our method on top of a CoT-vector model as a check that the gains stack.
Proposed Experiments
The decisive first experiment is the branch-point correlation diagnostic on PrOntoQA 3-hop and 5-hop fictional ontologies. It is cheap and determines whether the rest is worth running. Main accuracy experiments then cover PrOntoQA strict proof accuracy, GSM8K, AQuA, and StrategyQA on all three models, with $b \in \{2,4,8\}$ and both random and clustered buckets. We report accuracy-versus-compute curves and the fraction of steps that were contested, which shows whether compute actually concentrates on a few hard steps, as the hardest-step bound predicts.
On PrOntoQA we also report the rate of wrong choices at the first branch point, which checks the mechanism directly rather than only through accuracy. A secondary run on List Function and SCAN from the Curse of CoT suite asks whether short-bucket CoT narrows the gap to direct answering. Direct answering is included there as a baseline, because it may simply remain best.
The outcome that would falsify the proposal is simple: bucket errors at branch points correlate as strongly as sample errors, and terminal bucket voting matches step-level voting at equal compute.
References
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
- Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMs
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
- Reliable Chain-of-Thought via Prefix Consistency
- Exploring Demonstration Ensembling for In-context Learning
- Boosted Prompt Ensembles for Large Language Models
- An In-Context Learning Theoretic Analysis of Chain-of-Thought
- The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learning
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs
- Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic