When a language model drafts an answer and a panel of other models checks it, the system has to decide after each check whether to accept the draft, revise it, or ask one more checker. Current systems make this decision badly in two opposite ways. Some consult every checker on every round, so latency grows with rounds times checkers. Others stop as soon as a few checkers agree, which assumes their agreement is independent evidence. It often is not. Across hundreds of models, pairs that are both wrong pick the same wrong answer about 60% of the time on HELM, against roughly 33% by chance (Correlated Errors in Large Language Models). Agreement between two models from the same lineage is therefore worth much less than agreement between two unrelated ones.
We propose to make this discount explicit and use it to decide when to stop. On a labeled calibration set, we measure how often each pair of checkers fails together, and how that joint failure rate changes with problem difficulty. At run time, checkers are queried one at a time. After each verdict, the system updates its estimate of whether the current draft is correct, using that dependence model. It stops once this estimate crosses a threshold chosen so the rate of accepting wrong answers stays below a target. Easy problems should end after one or two cheap, unrelated checkers. Problems where checkers tend to share a mistake should be escalated to more diverse checkers or sent back for revision. The claim is that one correlation-corrected stopping rule both cuts latency and prevents the false consensus that makes current early stopping unsafe.
Core Claim
The latency of multi-model refinement and its vulnerability to correlated failure are the same problem seen from two sides. Both come down to how much evidence a given set of agreeing verdicts provides. A stopping rule that treats verdicts as independent stops too early exactly on the problems where checkers share a blind spot. A rule that ignores agreement altogether never stops early.
We claim that a sequential test with a dependence model fit on labeled calibration problems will do two things. First, it will match the accuracy of the strongest full-budget verification method, ExeCRE, at a fraction of its wall-clock cost. Second, it will keep its false-accept rate at the target, while an independence-assuming sequential rule, given the same checkers, will overshoot that target. The overshoot should be concentrated on problems where two or more model families make the same wrong prediction.
Where the Pieces Come From
Several papers motivate the gap by showing that agreement is not independent evidence. Kim et al. show error agreement is structured by shared provider, architecture and scale, which together explain 34% to 62% of its variance. They also find an "accuracy paradox": stronger models have more correlated residual errors. Chen et al. (Are More LLM Calls All You Need?) prove that majority voting over repeated calls to one model converges to the wrong answer on hard queries. On MMLU Physics, accuracy peaks at 10–20 calls and then falls. Hamidieh et al. (Complementing Self-Consistency with Cross-Model Disagreement) show that disagreement across model families catches "confidently wrong" answers that self-consistency misses. They also warn that this signal collapses when the peers share training gaps. None of these papers turns the observation into a stopping decision inside a refinement loop.
Other work constrains the design. CARE (CARE: Confounder-Aware Aggregation) models judges as linked by shared hidden biases and separates quality from those biases without labels. It relies on the assumption that true quality is the dominant shared factor, which can invert when a shared bias is stronger. We borrow its dependency-graph view of judges but give up label-freeness: we fit the dependence structure on labeled calibration problems, precisely so that we do not rely on that assumption. CARE is also an offline aggregator over a full panel, whereas we query checkers sequentially and stop early.
SRLA (Game-Theoretic Multi-LLM Routing) up-weights models whose errors point in different directions from the others ("leverage scores"). It does this for single-shot multiple-choice classification only. Our checker-ordering rule plays the same role, rewarding information that is orthogonal to what has already been gathered, but in open-ended code checking with revision.
Two further papers supply parts of the method. ExeCRE (ExeCRE) estimates the reliability of candidate reference programs from their execution agreement on about 1,000 synthetic inputs. It uses a Dawid–Skene latent-class model, which assumes the programs err independently given the truth. ExeCRE is our strongest baseline and the source of our verdict mechanism. It also illustrates both failure modes we target: it cannot detect a bug shared by all references, and it spends about 26 seconds per problem with no adaptive budget.
DEER (Dynamic Early Exit in Reasoning Models) shows that stopping early can improve accuracy by avoiding overthinking. Its stopping signal is a single model's own confidence, which is the signal Hamidieh et al. show fails on confident hallucinations. Critique panels such as N-Critics and Kumbhar et al. show that diverse critics catch complementary errors, but they consult every critic every round with no stopping rule.
Research Program
The main setting is code generation, because hidden tests give ground truth and allow false-accepts to be measured exactly. A generator model writes a draft. Each checker is a different-family model that writes its own reference solution. Its verdict is whether the draft and the reference agree on a small batch of generated inputs: a few dozen, not a thousand.
The checker pool deliberately includes pairs of varying relatedness: two sizes from one family, and several unrelated families. This makes the dependence structure non-trivial and testable.
The dependence model is a small log-linear model over checker verdicts, given whether the draft is correct and a cheap difficulty feature $z$ (the generator's mean token entropy on the draft). The key parameters are exactly Kim et al.'s quantity: how likely each checker pair is to agree when both are wrong. We fit this model on older LiveCodeBench problems with hidden tests.
At run time, after consulting a set $S$ of checkers with verdicts $a_S$, the system accepts the draft when
$$\hat P(\text{draft correct} \mid a_S, z) \ge \tau_\alpha,$$
where $\hat P$ is the dependence model's posterior and $\tau_\alpha$ is a threshold chosen on a held-out calibration split. The threshold is set so the empirical rate of accepting incorrect drafts is at most $\alpha$. This is a standard split-calibration argument, so the guarantee holds only as far as test problems resemble calibration problems.
The system sends the draft back for revision when the posterior falls below a lower threshold. Otherwise it queries the next checker. That checker is chosen to maximize expected reduction in decision uncertainty per second of expected latency. Under this rule, a checker highly correlated with those already consulted is naturally deprioritized.
Revision prompts include only the failing inputs from dissenting checkers, which keeps critiques short. The loop is capped at a fixed maximum number of rounds.
A secondary setting tests whether the idea transfers beyond code. It covers GSM8K with a Program-of-Thought pipeline, where the model writes code whose execution gives the answer (the ExeCRE setting), and HotpotQA with answer-level verdicts. On HotpotQA, agreement is judged by normalized exact match, not embedding similarity. Hamidieh et al.'s embedding-based disagreement misses swapped entities and digits, and exact match avoids that weakness.
Baselines
The naive reference point is the generator's single-pass output, with no checking or revision. Next is fixed-round self-repair using the generator's own tests. LiveCodeBench itself reports that GPT-4-Turbo rises from 24.5% to 36.9% with interpreter-feedback self-repair (LiveCodeBench). We rerun this with our generator rather than quoting it.
The strongest existing method is ExeCRE, run with its full input budget and reliability threshold. On LiveCodeBench v6 (182 problems, January–May 2025), it reports Pass@1 improving from 62.7% to 72.8% for GPT-5.2. It also reports that correct-code-unnecessarily-revised cases fall from 113.2 to 14.0, at about 26 seconds per problem (ExeCRE). We rerun ExeCRE with the same generator, the same checker pool as its source of reference programs, the same maximum rounds and the same hardware. We also run a variant with its input budget cut to match our median wall-clock time. Beating ExeCRE means one of two things: matching its Pass@1 within one standard error at no more than half its median latency, or exceeding its Pass@1 at matched latency, with unnecessary revisions no higher.
We also run a full-panel critic council in the style of N-Critics, which consults every checker every round and takes a majority vote. Separately, we run the same full panel aggregated with CARE-Tensor instead of our supervised dependence model. This tests whether label-free confounder separation is already enough.
A DEER-style single-model confidence exit is included for its latency profile. Chen et al.'s fixed-budget vote, with the call budget tuned on calibration data, represents the best non-adaptive allocation.
The ablation that isolates our key ingredient is the identical sequential procedure with the dependence model replaced by an independence assumption. It uses the same per-checker accuracies, the same ordering rule and the same calibration of $\tau_\alpha$. Every method gets the same tuning budget on the same calibration split. All methods report Pass@1, false-accept rate among accepted drafts, unnecessary revisions, tokens and median/95th-percentile wall-clock time.
Why not just…?
Why not just stop when two or three models from different families agree? This is the real threat. If the checker pool is chosen to be diverse enough, correlated agreement may be rare, and a simple unanimity rule may be as good. Kim et al.'s 60% conditional agreement and the accuracy paradox suggest it will not be, especially for strong checkers. The independence-assuming sequential ablation above tests this directly. If its false-accept rate matches ours on the correlated-error problems, the dependence model is unnecessary, and we would report that result.
Why not just run ExeCRE with fewer inputs? Shrinking the input budget cuts latency, but it does not address a bug shared by all reference programs. Dawid–Skene treats references as independent, so a shared bug reads as strong agreement. The matched-latency ExeCRE baseline measures the latency side of this. Stratifying by problems where several families produce the same wrong output measures the shared-bug side.
Why not just use the generator's own confidence to stop, as DEER does? Hamidieh et al. show that intra-model signals collapse on confident hallucinations. Their cross-model term improves AUROC by 0.15 on HotpotQA. The DEER-style baseline will show whether this carries over to code. We expect it to be fast but to have a false-accept rate well above target.
Why not just use CARE, which needs no labels? CARE's identifiability rests on quality being the dominant shared factor, an assumption Kim et al.'s findings put in doubt for strong models. The full-panel CARE baseline tells us whether this matters in practice. If CARE matches our accuracy and its full-panel cost is acceptable, the case for labeled calibration narrows to latency alone.
Decisive Checks
The central test uses held-out problems stratified by whether at least two checker families produce the same wrong output. On that stratum, the independence rule should exceed its false-accept target while ours stays near $\alpha$. On the complementary stratum, the two rules should behave almost identically. If there is no gap on the correlated stratum, the core claim is false.
The second test replaces the checker pool with stronger models of the same families and refits only the dependence model. The accuracy paradox predicts that the independence rule's false-accepts will rise while ours stay calibrated, at the cost of more checker queries on hard problems.
The third test is the latency–accuracy frontier. We plot Pass@1 against median wall-clock time for all methods, and the proposal succeeds only if its curve dominates ExeCRE's at some latency level.
Finally, the GSM8K and HotpotQA replications test whether pairwise failure correlations measured in code also predict false consensus when answers are short and non-executable. ExeCRE's reported Program-of-Thought GSM8K gain for LLaMA-3.1-8B (75.16% to 78.85%) is the reference point there.
References
- Correlated Errors in Large Language Models
- Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems
- Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
- Game-Theoretic Multi-LLM Routing for Safer Agents in the Wild
- ExeCRE: Execution-Consistency Guided Reliability Estimation for Self-Correcting Code Generation
- Dynamic Early Exit in Reasoning Models
- Enhancing Large Language Models with Ensemble of Critics for Mitigating Toxicity and Hallucination
- Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code