TL;DR
This paper formalizes agentic LLM workflows—specifically sample-and-select committee methods—as a form of inference-time "boosting" for reasoning models. The authors prove that while sampling more candidates improves the chance of generating a correct answer, a system cannot reliably select the right answer without an external local soundness signal (like code execution or test results). Remarkably, using this framework to orchestrate a weak model (GPT-5.4 nano) allows it to match the performance of frontier models like Gemini 3 Pro on software engineering tasks, up until it hits a hard ceiling imposed by the model's inherent "blind spots."
What Problem Does This Paper Address?
A popular trend in AI engineering is using "committees" or multi-agent workflows to improve LLM performance at inference time. While empirical results show that "more agents help," we lack a rigorous theoretical understanding of why and when these compound systems succeed or fail.
In traditional supervised machine learning, "boosting" combines multiple weak predictors to create a strong one by checking them against known training labels. In multi-step reasoning tasks (like coding or theorem proving), however, the system doesn't have a global label for intermediate steps. It must generate a local move, figure out if that move is helpful, and avoid letting small errors cascade into a completely wrong final answer.
The authors tackle a fundamental question: Can a committee of weak reasoning-model calls truly match the performance of frontier models, and what defines the mathematical limits of this approach?
Methods
The authors formalize verifier-backed agent systems as a bounded-depth search over partial states and break the problem down into four distinct quantities:
- Proposal Coverage: Does a good move appear in the sampled pool?
- Local Identifiability: Can the system recognize a good move when it sees it?
- Progress Depth: Do local choices successfully compose into a full solution?
- Diversity: Do additional model calls escape different failure modes?
To operationalize this, they introduce the Committee Protocol ($\Pi_{k,m,r}$), which executes three distinct steps at each reasoning state:
- Propose ($k$): Sample $k$ candidate actions from a weak proposer model.
- Critique ($m$): Run $m$ independent critic calls per candidate to filter out obviously flawed or refutable options.
- Compare ($r$): Use pairwise comparators to rank the surviving candidates, aggregating votes using a Copeland-style tournament rule to pick the winner.
Datasets
The authors evaluate their framework empirically on SWE-bench Verified, a dataset consisting of 500 real-world software engineering tasks pulled from GitHub. Success is determined by whether a generated code patch passes a suite of held-out, hidden unit tests.
Key Insights and Ideas
- Coverage Does Not Imply Identifiability: The authors mathematically prove a black-box separation theorem showing that simply being able to generate a correct solution with some probability does not give a model the intrinsic ability to criticize or rank those solutions. Reliable boosting requires an independent, local soundness signal—such as code execution, type checking, or test verification.
- The Blind-Spot Ceiling: Sampling more candidates reduces ordinary statistical noise, but it cannot overcome "shared blind spots." If a weak model assigns a 0% probability to the specific reasoning move required for a hard task, sampling it a million times will never surface the correct answer. Thus, "oracle best-of-$k$" performance eventually plateaus at a hard ceiling determined by these blind spots.
- Selection, Not Generation, is the Bottleneck: A massive amount of frontier-level capability is already latent within weak models. The primary challenge isn't getting the weak model to generate the right answer across multiple attempts; it's building a selection harness capable of finding it in the pile of candidates.
Key Results
The authors put their theory to the test by using a weak model, GPT-5.4 nano, as the proposer on SWE-bench Verified:
- Standalone Performance: A single one-shot proposal from
GPT-5.4 nanosolved 67.0% of tasks. - Orchestrated Performance: By generating $k=8$ proposals and applying their critic-comparator harness, the exact same nano model achieved a solve rate of 76.4%.
- Matching the Frontier: This orchestrated nano model successfully matched the standalone performance of much larger frontier models, including
Gemini 3 ProandClaude Opus 4.5 Thinking(both at 76.4%).
Ablation studies (Figure 3) demonstrated that critics and comparators are highly complementary: critics function as a coarse filter to eliminate bad moves, while comparators act as a fine-grained ranker for plausible alternatives.
Crucially, failure analysis (Figure 4) revealed that once the orchestration framework was applied, the remaining errors were almost entirely due to proposal-coverage failures (the blind-spot ceiling) rather than selection errors.
Limitations and Future Work
- Dependency on Local Signals: This boosting mechanism relies heavily on domains that provide clear, local soundness signals (e.g., compilers, test suites, or proof checkers). It is less clear how to apply this to open-ended or subjective text generation tasks where intermediate steps cannot be mechanically verified.
- Resource Trade-offs: While using a weaker model saves on base weights, running multiple proposal, critic, and comparator calls introduces significant inference-time costs, latency, and system complexity that must be weighed against simply using a single large model call.
- Breaking the Ceiling: Because the system is bounded by the proposer's blind spots, future work must focus on increasing proposer diversity (e.g., via different prompts, tools, or model architectures) rather than just building sharper selection tools.
Summary updated