Agentic Systems as Boosting Weak Reasoning Models

Varun Sunkaraneni, Pierfrancesco Beneventano, Riccardo Neumarker, Tomaso Poggio, Tomer GalantiarXiv 2026 · arXiv

TL;DR

This paper formalizes agentic LLM workflows—specifically sample-and-select committee methods—as a form of inference-time "boosting" for reasoning models. The authors prove that while sampling more candidates improves the chance of generating a correct answer, a system cannot reliably select the right answer without an external local soundness signal (like code execution or test results). Remarkably, using this framework to orchestrate a weak model (GPT-5.4 nano) allows it to match the performance of frontier models like Gemini 3 Pro on software engineering tasks, up until it hits a hard ceiling imposed by the model's inherent "blind spots."

What Problem Does This Paper Address?

A popular trend in AI engineering is using "committees" or multi-agent workflows to improve LLM performance at inference time. While empirical results show that "more agents help," we lack a rigorous theoretical understanding of why and when these compound systems succeed or fail.

In traditional supervised machine learning, "boosting" combines multiple weak predictors to create a strong one by checking them against known training labels. In multi-step reasoning tasks (like coding or theorem proving), however, the system doesn't have a global label for intermediate steps. It must generate a local move, figure out if that move is helpful, and avoid letting small errors cascade into a completely wrong final answer.

The authors tackle a fundamental question: Can a committee of weak reasoning-model calls truly match the performance of frontier models, and what defines the mathematical limits of this approach?

Methods

The authors formalize verifier-backed agent systems as a bounded-depth search over partial states and break the problem down into four distinct quantities:

  1. Proposal Coverage: Does a good move appear in the sampled pool?
  2. Local Identifiability: Can the system recognize a good move when it sees it?
  3. Progress Depth: Do local choices successfully compose into a full solution?
  4. Diversity: Do additional model calls escape different failure modes?

To operationalize this, they introduce the Committee Protocol ($\Pi_{k,m,r}$), which executes three distinct steps at each reasoning state:

  • Propose ($k$): Sample $k$ candidate actions from a weak proposer model.
  • Critique ($m$): Run $m$ independent critic calls per candidate to filter out obviously flawed or refutable options.
  • Compare ($r$): Use pairwise comparators to rank the surviving candidates, aggregating votes using a Copeland-style tournament rule to pick the winner.
Figure 2: One step of the committee protocol.

Datasets

The authors evaluate their framework empirically on SWE-bench Verified, a dataset consisting of 500 real-world software engineering tasks pulled from GitHub. Success is determined by whether a generated code patch passes a suite of held-out, hidden unit tests.

Key Insights and Ideas

  • Coverage Does Not Imply Identifiability: The authors mathematically prove a black-box separation theorem showing that simply being able to generate a correct solution with some probability does not give a model the intrinsic ability to criticize or rank those solutions. Reliable boosting requires an independent, local soundness signal—such as code execution, type checking, or test verification.
  • The Blind-Spot Ceiling: Sampling more candidates reduces ordinary statistical noise, but it cannot overcome "shared blind spots." If a weak model assigns a 0% probability to the specific reasoning move required for a hard task, sampling it a million times will never surface the correct answer. Thus, "oracle best-of-$k$" performance eventually plateaus at a hard ceiling determined by these blind spots.
  • Selection, Not Generation, is the Bottleneck: A massive amount of frontier-level capability is already latent within weak models. The primary challenge isn't getting the weak model to generate the right answer across multiple attempts; it's building a selection harness capable of finding it in the pile of candidates.

Key Results

The authors put their theory to the test by using a weak model, GPT-5.4 nano, as the proposer on SWE-bench Verified:

  • Standalone Performance: A single one-shot proposal from GPT-5.4 nano solved 67.0% of tasks.
  • Orchestrated Performance: By generating $k=8$ proposals and applying their critic-comparator harness, the exact same nano model achieved a solve rate of 76.4%.
  • Matching the Frontier: This orchestrated nano model successfully matched the standalone performance of much larger frontier models, including Gemini 3 Pro and Claude Opus 4.5 Thinking (both at 76.4%).
A committee of GPT-5.4 nano calls reaches much stronger models.

Ablation studies (Figure 3) demonstrated that critics and comparators are highly complementary: critics function as a coarse filter to eliminate bad moves, while comparators act as a fine-grained ranker for plausible alternatives.

Scaling with the number of proposals and orchestration components.

Crucially, failure analysis (Figure 4) revealed that once the orchestration framework was applied, the remaining errors were almost entirely due to proposal-coverage failures (the blind-spot ceiling) rather than selection errors.

Failure decomposition by benchmark category.

Limitations and Future Work

  • Dependency on Local Signals: This boosting mechanism relies heavily on domains that provide clear, local soundness signals (e.g., compilers, test suites, or proof checkers). It is less clear how to apply this to open-ended or subjective text generation tasks where intermediate steps cannot be mechanically verified.
  • Resource Trade-offs: While using a weaker model saves on base weights, running multiple proposal, critic, and comparator calls introduces significant inference-time costs, latency, and system complexity that must be weighed against simply using a single large model call.
  • Breaking the Ceiling: Because the system is bounded by the proposer's blind spots, future work must focus on increasing proposer diversity (e.g., via different prompts, tools, or model architectures) rather than just building sharper selection tools.

Summary updated

Abstract

Can a committee of weak reasoning-model calls reach the performance of much stronger models? We study verifier-backed committee search as inference-time boosting for reasoning language models. The mechanism is not simply that ``more agents help'': samples expose latent correct solutions, while critics and comparators must recover them without access to the hidden verifier. We formalize this view by separating proposal coverage, local identifiability, progress, and diversity. We prove that coverage can be amplified by repeated sampling, but cannot by itself create useful critics or comparators; reliable amplification requires an additional local soundness signal, such as execution, proof checking, type checking, tests, or constraint solving. We give rank-based bounds showing when local selection errors compose into reliable trajectories, and characterize the proposer-side ceiling: oracle best-of-\(k\) converges only to the mass of task slices on which the proposal system assigns nonzero useful probability. Empirically, on SWE-bench Verified, a single \texttt{GPT-5.4 nano} proposal solves \(67.0\%\) of tasks. Using the same nano model, our critic--comparator orchestration reaches \(76.4\%\) with \(k=8\) proposals, matching the standalone performance of \texttt{Gemini 3 Pro} and \texttt{Claude Opus 4.5} Thinking and approaching the \(79.0\%\) oracle best-of-\(8\) upper bound. Thus, many correct patches are already present in weak-model proposal pools; the main challenge is selecting them. The remaining failures are mostly proposal-coverage failures, indicating shared blind spots that stronger selection alone cannot close.
BibTeX
@misc{sunkaraneni2026agentic,
  author = {Varun Sunkaraneni and Pierfrancesco Beneventano and Riccardo Neumarker and Tomaso Poggio and Tomer Galanti},
  title = {{Agentic Systems as Boosting Weak Reasoning Models}},
  year = {2026},
  eprint = {2605.14163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.14163},
}
Figure