Two popular ways to make language models more reliable are expensive in different directions. Ensembling runs several models, prompts, or samples on the same query and combines them, which multiplies compute by the number of members. Iterative refinement has a model draft an answer, critique it, and rewrite it, which multiplies latency by the number of rounds, because each rewrite must wait for the previous one. Systems that do both pay roughly members times rounds. Almost all of that cost is wasted on text that never changes. When five candidate programs agree on forty of fifty lines, a refinement loop that regenerates all fifty lines every round spends most of its sequential decoding re-deriving the consensus.
This note proposes disagreement-localized refinement. The ensemble's own disagreement decides three things: where to refine, whether to refine at all, and when to stop. Candidate outputs are aligned at the level of lines or reasoning steps. Spans on which most members agree are frozen and processed in one parallel pass. Only the contested spans are regenerated, and each is conditioned on the frozen context and on the competing versions of that span. The loop ends when no contested spans remain or when they stop changing. The claim is that the latency of combined ensembling and refinement can be made to scale with the size of the disagreement rather than with output length times rounds, at equal accuracy. A second claim is that cross-member disagreement is a better locator of errors than a single model's self-critique.
Core Claim and What Is New
The closest existing systems each handle one half of the compound cost.
SpecEM ensembles several LLMs segment by segment. Every model drafts a fixed-length chunk, about 10 tokens, and all models cross-verify the chunks in one forward pass using modified attention masks. It never revisits earlier content, and the segment length is fixed rather than set by where models disagree. Let's Revise Step-by-Step (ReLoc) keeps a pool of candidate programs and revises them by local search under a learned revision-distance reward. Its revisions still rewrite whole programs over sequential rounds. PREFER refines a prompt ensemble offline, and its own authors note that running the ensemble at inference remains costly. Scaling Up, Speeding Up (SpecTTS-Bench) shows that refinement rounds repeat a lot of text. Its suffix-automaton drafter gets faster in later rounds by copying earlier turns, but it is lossless token-level speculation on a single model and leaves the structure of the loop unchanged.
Our proposal differs from all of them in one specific way. It treats the alignment of ensemble candidates as a map of where errors probably are. It then changes what the refinement loop regenerates, rather than speeding up the generation of an unchanged loop.
The supporting evidence for the gap comes from two open problems in the file. Iterative editing drifts and lacks clean stopping signals (Extrapolative Controlled Sequence Generation via Iterative Refinement). Weak ensemble members clutter verification until their weights adapt (SpecEM's limitation). Agreement-based freezing addresses drift directly, because consensus spans cannot be edited. It also supplies a stopping signal: the contested set is empty, or it is the same as last round.
Research Program
The system has four components.
Ensemble. The primary setting is cheap: $K$ temperature samples plus a few prompt variants from one open model, sharing the prompt's attention cache. A secondary setting uses three heterogeneous 7B-class models to test whether the idea survives different tokenizers. Alignment operates on text, so tokenizer mismatch matters less here than it does for the Jacobi-style drafters in xPress.
Aligner. The aligner picks the medoid candidate as a skeleton and aligns the other candidates to it.
- For code, alignment is at the line and abstract-syntax-tree level.
- For math, alignment is at the step level, matching normalized intermediate quantities.
A span is frozen when at least $\tau K$ members agree on it. Every other span becomes a hole.
Infilling reviser. The same model receives the skeleton with holes and, for each hole, the competing member versions. Holes are filled in parallel as a batch, under an independence approximation. Inconsistencies between holes simply reappear as disagreement in the next round.
Member pruning. Members whose versions of contested spans are repeatedly overruled are dropped from later rounds. This uses a multiplicative-weights rule in the spirit of SpecEM, but it removes those members' compute instead of only downweighting their votes.
Cost model. A full-rewrite loop has sequential decoding steps on the order of $R \cdot L$, where $R$ is the number of rounds and $L$ is the output length in tokens. The localized loop costs roughly
$$ T_{\text{seq}} \approx T_{\text{draft}} + \sum_{r=1}^{R'} \max_{h \in H_r} |h|, $$
where $T_{\text{draft}}$ is the cost of the initial parallel drafts, $H_r$ is the set of holes in round $r$, $|h|$ is the token length of hole $h$, and $R'$ is the number of rounds actually run under the disagreement stopping rule. The maximum appears because holes are filled in parallel. Frozen spans contribute only prefill, which is parallel across tokens. The expected gain is therefore the ratio of output length to contested-span length. This form comes directly from counting sequential decoding steps. The gain should be large on code and multi-step math, where candidates share most of their structure. It should be small on open-ended prose.
Analysis. Alongside the system, we will test the locator directly. Is the cross-member contested set a more precise error locator than spans a trained critic flags? The comparison critic is in the style of Critique-RL. Precision and recall are measured against the lines changed in a minimal correct fix (code) or the first erroneous step (math).
Baselines
Single pass. The naive reference is one greedy decode from the base model. It sets the latency floor and the accuracy floor. On MBPP, Meta-Reflection reports LLaMA-3.1 zero-shot Pass@1 of 58.8%. A matching-model comparison will rerun this.
Majority voting. Parallel sampling with majority voting or execution-based selection, which requires no refinement. Critique-RL reports that critique-and-refine reaches "up to a 5x efficiency advantage" over majority voting on MATH with Qwen2.5-7B, along with a 9.02% in-domain gain. That makes this baseline a meaningful bar on the other side.
Full-rewrite refinement on the same ensemble. - The ReLoc hill-climbing variant, rerun with our base model. ReLoc reports LiveCodeBench Pass@1 improving "from 33.8% to 38.4%" over the strongest improvement-based baseline, with 37% fewer tokens than tree search. - Critique-RL's response-critique-refine cycle, applied to math.
These are the strongest competitors because they reach the accuracy we aim to keep.
Lossless acceleration of full rewrites. The same full-rewrite loop, accelerated with SpecTTS-Bench's hybrid suffix-automaton drafter, run from the released SpecTTS-Bench code. The file reports round-two speedup gains of 33–45% for this approach. It is the most direct latency competitor.
Single-pass student. A student distilled from the ensemble's refined outputs, following Teaching Language Models to Self-Improve by Learning from Language Feedback, which reports gains of 3.7 to 4.0 points on GSM8K, BBH and TydiQA from training on critique-refinement data.
Fairness rules for all comparisons. - Every method uses the same base model or models, the same $K$, and the same hardware with batched serving. - Every method gets an equal token budget for tuning hyperparameters such as $\tau$, the number of rounds, and the vote threshold. - Results are reported as accuracy against median and 95th-percentile wall-clock latency, and against total generated tokens.
What counts as winning. We win if we match the full-rewrite loop's accuracy within noise at lower median latency than its suffix-automaton-accelerated version, or if we achieve higher accuracy than majority voting at equal latency.
Ablations. Each removes one ingredient:
- Localization off: full rewrite, but keeping our stopping rule.
- Fixed rounds instead of disagreement-based stopping.
- Pruning off.
- Random spans of matched length as holes. This isolates whether disagreement, rather than mere shortening, carries the benefit.
- Critic-flagged spans replacing disagreement spans.
Why not just…?
Why not just accelerate the full-rewrite loop with speculative decoding? This is the real threat. When candidates sit in the reviser's context, a suffix automaton can already copy consensus text cheaply, as SpecTTS-Bench's progressive acceleration shows. There are two reasons we expect localization to still win. First, speculative acceptance proceeds a few tokens per step and still walks the whole output each round, whereas frozen spans cost one prefill. Second, SpecTTS-Bench reports that n-gram acceleration degrades as temperature rises, and ensembles depend on high-temperature diversity. The two methods also compose, and we will run them together, since holes can be speculatively decoded too. If localization adds no wall-clock benefit over this baseline at temperature 0.7–1.0, the systems claim fails. Only the error-localization claim would then remain.
Why not just take the majority vote and skip refinement? Voting has no latency multiplier. On free-text outputs it may genuinely be the better trade-off. Critique-RL's 5x efficiency result suggests refinement wins on math. ReLoc shows that best-of-N wastes samples when pass rates are sparse. Our accuracy-latency curves will show where the crossover lies for each benchmark.
Why not just distill the ensemble-plus-refinement pipeline into one model? Distillation removes test-time cost entirely and is the right answer if it keeps the accuracy. Distilling Knowledge for Search-based Structured Prediction reports that the distilled student still trails global search methods, and the gap between a single-pass student and test-time search is exactly what we measure. We report the student as a separate point on the curve, not as a strawman.
Why not just route each query to one expert, or average weights? Routing helps only if some single member is reliably right. Beyond Monolithic Scaling finds that aggregation beats the best single expert on hard queries. Model soups requires a shared initialization and does not apply to sample or prompt ensembles at all.
Proposed Experiments and Decisive Checks
Benchmarks. - Code: MBPP, HumanEval and LiveCodeBench, evaluated by execution Pass@1. - Math: GSM8K and MATH-500, evaluated by answer accuracy. - Negative control: AlpacaEval 2.0, where free text should give poor alignment and small gains.
Reported results. - Accuracy–latency Pareto curves. - Mean contested fraction per round. - Number of rounds actually used. - Error-locator precision and recall.
The file contains no reported wall-clock results for ensemble-plus-refinement loops on GSM8K or MBPP, so all latency numbers will be measured fresh under identical serving.
Decisive check 1: is disagreement a good error locator? If contested spans do not locate true errors better than critic-flagged spans, the core premise is wrong. The random-span ablation is also part of this check. If random spans of the same length match our accuracy, the benefit comes from shortening alone, not from disagreement.
Decisive check 2: does it beat accelerated full rewrites at temperature? If our method does not beat the suffix-automaton-accelerated full-rewrite loop on wall-clock at sampling temperatures of 0.7 and above, the latency contribution reduces to existing speculative decoding.
Decisive check 3: does the stopping rule cause drift or premature freezing? Tracking accuracy per round will show whether the disagreement stopping rule reduces semantic drift relative to fixed-round refinement, or whether it instead locks in confident consensus errors. This failure mode is most likely when all members share a bias, so we will measure it separately on the heterogeneous-model ensemble.
References
- SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback
- Let's Revise Step-by-Step: A Unified Local Search Framework for Code Generation with LLMs
- PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
- Extrapolative Controlled Sequence Generation via Iterative Refinement
- xPress: Parallel Refinement for Diffusion Drafters in Speculative Decoding
- Critique-RL: Training Language Models For Critiquing Through Two-Stage Reinforcement Learning
- Meta-Reflection: A Feedback-Free Reflection Learning Framework
- Teaching Language Models to Self-Improve by Learning from Language Feedback
- Distilling Knowledge for Search-based Structured Prediction
- Beyond Gemini-3-Pro: Revisiting LLM Routing and Aggregation at Scale
- Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time