TL;DR
Researchers at NVIDIA have developed a framework to predict how well a base LLM will perform as a coding agent before investing in expensive post-training. By identifying the "decisive step" in successful agent trajectories—the exact moment a code change flips a task from failing to passing—they created three probes that correlate strongly (Spearman $\rho > 0.9$) with final agentic performance on benchmarks like SWE-bench Verified.
What Problem Does This Paper Address?
Training a state-of-the-art coding agent is a high-stakes gamble. Developers must choose a "base" model (a raw, pre-trained checkpoint) to undergo agentic post-training (SFT/RLHF) without knowing if that model has the inherent potential to handle complex, multi-step repository edits.
Current evaluation methods fall into two traps:
- Too Hard: End-to-end benchmarks like SWE-bench are nearly impossible for base models because they lack the "agentic" formatting (like tool-calling) to even start the task.
- Too Simple: Benchmarks like HumanEval (single-function synthesis) don't capture the core difficulty of agentic coding: maintaining state across an evolving codebase.
As a result, a base model might look great on HumanEval but fail miserably as a post-trained agent. This paper seeks a "lookahead signal" to identify promising base models early.
Methods
The authors propose a trajectory-derived probing framework. They start by collecting successful trajectories from "frontier" models (like GPT-4 or similar) that have already solved tasks in an agentic harness.
The Decisive Step
The key innovation is identifying the Decisive Step ($T^*$). By replaying a successful trajectory and running the task's verifier (unit tests) after every code change, they find the first step where the cumulative patch causes the tests to pass. This specific action is labeled the "golden action."
The Three Probes
Once $T^*$ is identified, the base model is tested at that exact decision point using three methods:
- Decisive-Action BPB (Bits Per Byte): Measures the probability mass the base model assigns to the golden action. Using BPB makes this metric comparable across models with different tokenizers.
- Patch MCQ (Multiple Choice Question): The model is asked to choose the golden action from a set of distractors (plausible-looking edits that actually fail the verifier).
- Prefix-conditioned pass@K: The model is given the trajectory prefix up to $T^*$ and asked to generate its own solution. If any of the $K$ samples pass the verifier, it succeeds.
Datasets
The study evaluates ten pairs of base and post-trained models (including the Nemotron-3 family, Qwen-3.5, DeepSeek-V4, and Kimi-K2) across several benchmarks:
- SWE-bench Verified: The primary target for agentic performance.
- DeepSWE: A benchmark for original, long-horizon engineering tasks.
- SWE-Pro: A benchmark for enterprise-style coding tasks.
- Terminal-Bench 2.1: Focused on command-line workflows.
- Baseline Benchmarks: HumanEval, MBPP, RepoBench XFirst, and CRUXEval were used for comparison.
Key Insights and Ideas
- The Coverage Principle: A good base model doesn't need to be a perfect agent yet; it just needs to have "coverage"—assigning high probability to the correct behaviors that post-training can later sharpen.
- The Decisive Step is Essential: Probing the model at the decisive step is significantly more predictive than averaging performance across all steps of a trajectory. The decisive step filters out "noise" (like exploration or incidental edits) and focuses on the actual problem-solving moment.
- Joint Scoring Matters: For the MCQ probe, the authors found that presenting all options in a single context and reading the "answer-letter likelihood" was far more effective than scoring each option independently.
- HumanEval is a Poor Predictor: The paper confirms that standard synthesis benchmarks like HumanEval have a negative correlation with downstream agentic performance in some cohorts, making them potentially misleading for model selection.
Key Results
The proposed probes show remarkable predictive power. Across ten model families, the correlation between the base model's probe score and the post-trained model's SWE-bench performance was consistently high.
- Decisive-Action BPB achieved a Spearman $\rho$ of 0.964.
- Prefix-conditioned pass@K achieved a Spearman $\rho$ of 0.951.
- These metrics significantly outperformed the best-bounded baseline, RepoBench XFirst ($\rho = 0.830$).
Limitations and Future Work
- Dependency on Trajectories: The method requires a set of successful trajectories from a frontier model. While these are increasingly available, the probes are only as good as the trajectories they are built from.
- Computational Cost of pass@K: While BPB and MCQ are "static" and cheap to compute, the prefix-conditioned pass@K requires running the full verification harness (containers, unit tests), which is more resource-intensive.
- Generalization: While the framework works for coding, the authors suggest it could be applied to other agentic domains (like web browsing or tool use), but this remains to be tested.
Summary updated