Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models

Tan Yu, Alexander Bukharin, Khushi Bhardwaj, Jennifer Williams, Zirui Liu, Jonathan Lingjie Li, Soumye Singhal, Joseph Jennings, Sanjeev Satheesh, Yash Jain, Ashish Vaswani, Venkat Krishna Srinivasan, Matthew Papakipos, Hyunwoo Kim, Jian Zhang, Oleksii Kuchaiev, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jonathan Cohen, Jiantao JiaoarXiv 2026 · arXiv

TL;DR

Researchers at NVIDIA have developed a framework to predict how well a base LLM will perform as a coding agent before investing in expensive post-training. By identifying the "decisive step" in successful agent trajectories—the exact moment a code change flips a task from failing to passing—they created three probes that correlate strongly (Spearman $\rho > 0.9$) with final agentic performance on benchmarks like SWE-bench Verified.

What Problem Does This Paper Address?

Training a state-of-the-art coding agent is a high-stakes gamble. Developers must choose a "base" model (a raw, pre-trained checkpoint) to undergo agentic post-training (SFT/RLHF) without knowing if that model has the inherent potential to handle complex, multi-step repository edits.

Current evaluation methods fall into two traps:

  1. Too Hard: End-to-end benchmarks like SWE-bench are nearly impossible for base models because they lack the "agentic" formatting (like tool-calling) to even start the task.
  2. Too Simple: Benchmarks like HumanEval (single-function synthesis) don't capture the core difficulty of agentic coding: maintaining state across an evolving codebase.

As a result, a base model might look great on HumanEval but fail miserably as a post-trained agent. This paper seeks a "lookahead signal" to identify promising base models early.

Methods

The authors propose a trajectory-derived probing framework. They start by collecting successful trajectories from "frontier" models (like GPT-4 or similar) that have already solved tasks in an agentic harness.

The Decisive Step

The key innovation is identifying the Decisive Step ($T^*$). By replaying a successful trajectory and running the task's verifier (unit tests) after every code change, they find the first step where the cumulative patch causes the tests to pass. This specific action is labeled the "golden action."

A unified framework for identifying the decisive step and applying three predictive probes.

The Three Probes

Once $T^*$ is identified, the base model is tested at that exact decision point using three methods:

  1. Decisive-Action BPB (Bits Per Byte): Measures the probability mass the base model assigns to the golden action. Using BPB makes this metric comparable across models with different tokenizers.
  2. Patch MCQ (Multiple Choice Question): The model is asked to choose the golden action from a set of distractors (plausible-looking edits that actually fail the verifier).
  3. Prefix-conditioned pass@K: The model is given the trajectory prefix up to $T^*$ and asked to generate its own solution. If any of the $K$ samples pass the verifier, it succeeds.

Datasets

The study evaluates ten pairs of base and post-trained models (including the Nemotron-3 family, Qwen-3.5, DeepSeek-V4, and Kimi-K2) across several benchmarks:

  • SWE-bench Verified: The primary target for agentic performance.
  • DeepSWE: A benchmark for original, long-horizon engineering tasks.
  • SWE-Pro: A benchmark for enterprise-style coding tasks.
  • Terminal-Bench 2.1: Focused on command-line workflows.
  • Baseline Benchmarks: HumanEval, MBPP, RepoBench XFirst, and CRUXEval were used for comparison.

Key Insights and Ideas

  • The Coverage Principle: A good base model doesn't need to be a perfect agent yet; it just needs to have "coverage"—assigning high probability to the correct behaviors that post-training can later sharpen.
  • The Decisive Step is Essential: Probing the model at the decisive step is significantly more predictive than averaging performance across all steps of a trajectory. The decisive step filters out "noise" (like exploration or incidental edits) and focuses on the actual problem-solving moment.
  • Joint Scoring Matters: For the MCQ probe, the authors found that presenting all options in a single context and reading the "answer-letter likelihood" was far more effective than scoring each option independently.
  • HumanEval is a Poor Predictor: The paper confirms that standard synthesis benchmarks like HumanEval have a negative correlation with downstream agentic performance in some cohorts, making them potentially misleading for model selection.

Key Results

The proposed probes show remarkable predictive power. Across ten model families, the correlation between the base model's probe score and the post-trained model's SWE-bench performance was consistently high.

Correlations between the three probes and post-trained SWE-bench Verified performance.
  • Decisive-Action BPB achieved a Spearman $\rho$ of 0.964.
  • Prefix-conditioned pass@K achieved a Spearman $\rho$ of 0.951.
  • These metrics significantly outperformed the best-bounded baseline, RepoBench XFirst ($\rho = 0.830$).

Limitations and Future Work

  • Dependency on Trajectories: The method requires a set of successful trajectories from a frontier model. While these are increasingly available, the probes are only as good as the trajectories they are built from.
  • Computational Cost of pass@K: While BPB and MCQ are "static" and cheap to compute, the prefix-conditioned pass@K requires running the full verification harness (containers, unit tests), which is more resource-intensive.
  • Generalization: While the framework works for coding, the authors suggest it could be applied to other agentic domains (like web browsing or tool use), but this remains to be tested.

Summary updated

Abstract

How can we predict which base checkpoint is worth an expensive round of agentic post-training? End-to-end pass@$K$ tests whether successful behavior already appears in a base model's distribution, but it is a poor fit for agentic coding: many base checkpoints cannot reliably produce the well-formed tool invocation required to complete a task end-to-end. Single-shot or short-horizon tasks avoid these tool-calling failures by collapsing a multi-step interaction into a fixed prompt and a single patch, but they sidestep the core capability we care about: maintaining coherent state over many tool-using steps as the repository evolves. To bridge this gap, we treat successful post-trained agent trajectories as a lookahead signal of base-model potential. Replaying each trajectory and rerunning tests after every code-changing step identifies the decisive step: the first step whose cumulative patch flips the repository from failing to passing, certifying that the recorded action solves the task given the prior context. Motivated by a coverage principle for agentic traces, we build three screens at this step that do not require a base checkpoint to drive the harness from a cold start: (i) Decisive-Action BPB (bits per byte) measures the probability mass on the certified action, (ii) Patch MCQ tests the checkpoint's choice between that action and alternatives rejected by the same verifier, and (iii) prefix-conditioned pass@$K$ evaluates support for functionally-correct generations and credits any continuation that the tests accept. Across ten pairs of public base and post-trained models, all three screens rank the cohort in close agreement with post-trained SWE-bench Verified pass@$1$. As our methods need only a benchmark's successful trajectories and its verifier, they can be applied to turn future agentic coding benchmarks into base-model evaluations.
BibTeX
@misc{yu2026before,
  author = {Tan Yu and Alexander Bukharin and Khushi Bhardwaj and Jennifer Williams and Zirui Liu and Jonathan Lingjie Li and Soumye Singhal and Joseph Jennings and Sanjeev Satheesh and Yash Jain and Ashish Vaswani and Venkat Krishna Srinivasan and Matthew Papakipos and Hyunwoo Kim and Jian Zhang and Oleksii Kuchaiev and Markus Kliegl and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro and Jonathan Cohen and Jiantao Jiao},
  title = {{Before They Can Solve: Predicting Post-Training Coding-Agent Performance from Base Models}},
  year = {2026},
  eprint = {2610.10478},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.10478},
}
Figure