Evaluating a coding or terminal agent now means running it inside a sealed software environment, a container, for each task and checking the result with hidden tests. On Terminal-Bench a single attempt can run for up to two hours, make hundreds of API calls, and consume nearly 100 million tokens. Tasks also differ enormously in how expensive and how noisy they are. Some containers take minutes to build. Some reference solutions fail intermittently. Some tasks cost twenty times more tokens than others. Practitioners cut costs by running a hand-picked subset such as SWE-bench Lite or Verified, or a few hundred random items with one seed. Neither choice has any guarantee on how close the result lands to the full-benchmark score.
We propose to treat agent evaluation as a classical survey-design problem, with the execution environment supplying both the population structure and the cost model. The idea in two sentences: group tasks by measurable properties of their containers, such as build cost, run cost, and how often the known-correct solution itself fails on a clean rerun. Then use stratified sampling with a two-stage allocation rule that decides, within each group, how many distinct tasks to run and how many times to repeat each one. We claim this design gives a lower-error estimate of an agent's full-benchmark pass rate per container-hour than uniform subsampling or response-pattern methods like tinyBenchmarks. We also expect the gain to hold when the design is built on older agents and applied to newer ones, because strata defined by environment properties do not go stale the way item parameters learned from past model answers do.
Why this gap is real
Stratified subsampling of static benchmarks is well developed. tinyBenchmarks compares stratified random sampling, clustering by past model correctness ("anchor points"), and item response theory (IRT), a psychometric model that places each model's ability and each item's difficulty on a shared scale. Its gp-IRT estimator reaches an average error of 1.9% on MMLU with 100 examples. Efficient Safety Benchmarking via Item Response Theory applies adaptive and static IRT item selection to safety suites and keeps Spearman rank correlation with full rankings above 0.90. Both works assume that every item costs the same, that each item gives one noise-free binary answer per model, and that item parameters stay valid over time. tinyBenchmarks itself flags the last assumption as a weakness under model evolution. None of these assumptions holds for containerized agent tasks. These papers supply the estimators we compare against and partly reuse; they do not address the regime we target.
The execution-environment literature explains why the assumptions fail. A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests traces nondeterministic test outcomes to three sources: the system under test, the test infrastructure, and the external environment. GitTaskBench attributes 65% of agent failures to environment setup and dependency resolution. Establishing Best Practices for Building Rigorous Agentic Benchmarks finds that 70% of audited agent benchmarks violate task or outcome validity. These findings motivate us: part of the observed pass/fail variance comes from the environment rather than the agent, and it is concentrated in identifiable tasks.
Curation work constrains what we claim. Agentless built SWE-bench Lite-S by manual filtering. SWE-Bench Pro and Terminal-Bench build containerized suites by expert selection. Investigating Test Overfitting on SWE-bench and Saving SWE-Bench study validity on fixed subsets. None of them asks how to sample from a containerized suite under a compute budget with an error guarantee. That question is ours. We treat the full benchmark as the population and do not claim to fix its representativeness of real-world software work.
Core Claim
Let the benchmark be partitioned into strata $h$, each holding a fraction $W_h$ of the tasks. We draw $n_h$ tasks from stratum $h$ and run each $r_h$ times in fresh containers. The estimated pass rate is the weighted mean of per-stratum average outcomes. Its variance is
$$\mathrm{Var}(\hat p)=\sum_h \frac{W_h^2}{n_h}\left(\sigma_{b,h}^2+\frac{\sigma_{w,h}^2}{r_h}\right),$$
where $\sigma_{b,h}^2$ is the between-task variance of true success probability in stratum $h$ and $\sigma_{w,h}^2$ is the run-to-run variance of a single task, which comes from agent sampling randomness and environment flakiness. The cost of stratum $h$ is $n_h(c_{e,h}+r_h c_{a,h})$. Here $c_{e,h}$ is the one-time cost per task of building and validating its container, and $c_{a,h}$ is the cost of one agent run. Minimizing variance at fixed total cost gives the standard two-stage survey result:
$$r_h^\ast=\sqrt{\frac{\sigma_{w,h}^2\,c_{e,h}}{\sigma_{b,h}^2\,c_{a,h}}},\qquad n_h\propto \frac{W_h\sqrt{\sigma_{b,h}^2+\sigma_{w,h}^2/r_h^\ast}}{\sqrt{c_{e,h}+r_h^\ast c_{a,h}}}.$$
The formula captures an intuition that current practice ignores. Where containers are expensive to build and runs are noisy, repeat each task several times. Where runs are expensive and deterministic, spread the budget over more distinct tasks. Strata that are cheap and heterogeneous get more tasks than their share of the population. None of this is new statistics. What we add is measuring each quantity from the execution environment, and showing that these measurements are stable enough, and different enough across tasks, for the allocation to matter.
Research Program
The first component profiles each benchmark's environments. For every task in SWE-bench Verified (500 tasks), the SWE-Bench Pro public set (731), Terminal-Bench 2.0 (89), and GitTaskBench (54, via QuantaAlpha/GitTaskBench), we build the container from scratch and record build time and image size. We apply the reference solution ten times in fresh containers and record the gold-pass rate. This rate is a direct, agent-free measure of flakiness, which we split by failure type using the flaky-test taxonomy of Habchi et al. We also record test-suite runtime and static features such as repository, language, and patch size. Stratification uses only these agent-free features. We create strata by grouping tasks on build cost, gold-pass rate, and repository or category, choosing cut points on a held-out calibration panel.
The second component builds response data. We run a panel of scaffolds on every task with at least five seeds each: OpenHands (OpenHands), Agentless, and Terminus 2 on Terminal-Bench, each paired with several open-weight and API models. Every run logs its container-hours and tokens, and every run is labeled as either a setup/infrastructure failure or a verified agent failure from the harness logs. This yields a three-way array of agent × task × seed, with costs attached. That is the object missing from prior compression work, and we will release it.
The third component estimates $\sigma_b$, $\sigma_w$, and costs per stratum from a small pilot panel, applies the allocation above, and computes design-based confidence intervals. In a hybrid variant, we also fit a gp-IRT-style correction within strata, so that "stratified plus IRT" becomes a candidate and not only a competitor.
Baselines
The naive reference point is uniform random sampling of tasks with one run per task, at the same total container-hour budget as every other method, averaged over 200 random draws. Its error is exactly what a practitioner gets today from a quick subset. Beating it by a clear margin at matched cost is the minimum bar. The fixed curated subsets, SWE-bench Lite and Verified, serve as a second naive reference. They are cheap and widely used, but their error relative to the full SWE-bench score has never been reported with uncertainty, and we will measure it.
The strongest existing method is tinyBenchmarks' gp-IRT with anchor-point selection (tinyBenchmarks), plus the Fisher-information static subset selection from Efficient Safety Benchmarking via Item Response Theory. For the IRT methods we collapse seeds to majority pass/fail, which is how they would be used in practice, and also test a variant fit on the full seed-level data. They are fit on the same calibration agents with the same hyperparameter search budget as our design. Their items are charged at true per-task cost, so the comparison is per container-hour and not per item. Reported numbers exist only for static benchmarks: 1.9% average error on MMLU with 100 items and Spearman above 0.95 (tinyBenchmarks), and Spearman above 0.90 on safety suites (Efficient Safety Benchmarking via Item Response Theory). No reported result was found on Lacuna for IRT or stratified subsampling of SWE-bench, SWE-Bench Pro, Terminal-Bench, or GitTaskBench. We will produce these baseline numbers ourselves. We count our design as beating gp-IRT only if two conditions hold. At matched container-hours, its root-mean-square error of the estimated full pass rate must be at least 25% lower, with bootstrap intervals over agents not overlapping. And its rank correlation must be no worse.
The ablations isolate the key ingredients. We run the design with cost weighting removed, with $r_h$ fixed at 1, with strata built from correctness clusters instead of environment features, and with flakiness strata merged. If the environment-feature strata or the repeat allocation do not account for most of the gain, the central claim fails.
Why not just…?
Why not just use uniform sampling with more seeds? This is the practitioner's default. It is optimal when strata are homogeneous in cost and variance. Our environment profiling tests exactly that condition. If $\sigma_w$ and build cost barely vary across tasks, the formula collapses to uniform allocation and we will report that result. The existing evidence suggests heterogeneity is large: setup failures dominate GitTaskBench outcomes, and per-attempt cost on Terminal-Bench spans orders of magnitude.
Why not just apply tinyBenchmarks' gp-IRT to the pass/fail matrix? This is the real threat, and it may win in accuracy at a fixed number of items. It has three weaknesses in this regime. It ignores per-item cost, so it favors highly discriminative items that may be the most expensive. It treats a flaky environment failure as evidence about agent ability. And its parameters come from past agents' answers, which its authors note drift as models change. The temporal-split experiment below settles the question. If gp-IRT still wins at matched cost on newer agents, the hybrid variant becomes the recommendation, and the contribution narrows to the cost-and-rerun allocation.
Why not just fix the flaky environments first, following the agentic benchmark checklist? Hardening environments as Establishing Best Practices for Building Rigorous Agentic Benchmarks recommends is worthwhile, and we will do it where feasible. It does not remove the problem, for two reasons. First, the agents themselves sample stochastically, so $\sigma_w$ stays above zero even in perfectly deterministic containers. Second, cost heterogeneity is untouched. We also rerun the whole comparison on a de-flaked subset, keeping only tasks with a 100% gold-pass rate, to measure how much of our gain survives.
Proposed Experiments
The main experiment is a cost–error curve for each benchmark. The x-axis is container-hours plus token cost as a fraction of full evaluation. The y-axis is the root-mean-square error of the estimated pass rate, together with Spearman correlation across agents and the empirical coverage of nominal 95% intervals. Every method above appears on the same curve.
The decisive check is a temporal split. We calibrate every method only on agents built on models released before a cutoff date, then evaluate on agents released after it, including at least one model at the frontier of that benchmark. On SWE-Bench Pro, for example, GPT-5 reached 25.9% Pass@1 in SWE-Bench Pro. If environment-derived strata keep their advantage while IRT error grows, the drift argument holds. If both degrade equally, it does not.
The second check is interval coverage. Design-based intervals should cover the full-benchmark score close to 95% of the time. Stratified designs that ignore within-task variance should under-cover on flaky strata, and we will show whether they do.
The third check tests transfer. We ask whether an allocation tuned on SWE-bench Verified predicts good allocations on the multilingual and long-horizon suites. That result would tell benchmark maintainers whether to publish environment profiles alongside tasks as a standard artifact.
References
- tinyBenchmarks: evaluating LLMs with fewer examples
- Efficient Safety Benchmarking via Item Response Theory
- A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests
- GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Agentless: Demystifying LLM-based Software Engineering Agents
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Investigating Test Overfitting on SWE-bench
- Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents