Coding-agent benchmarks such as SWE-bench score an agent by running its patch against unit tests inside a container. That pass/fail outcome is assumed to measure the agent. Audits show it often does not. UTBoost found 345 false passes caused by a fragile log parser and thin tests, enough to reorder 29 leaderboard positions. Establishing Best Practices for Building Rigorous Agentic Benchmarks found that 70% of audited agent benchmarks violate basic validity standards. Both studies located these problems by hand, one bug at a time. We propose a statistical screen that, for every task in a benchmark, estimates how much of the variation in its outcome is explained by the patch the agent wrote, and how much is explained by the evaluation machinery and the machine it ran on.
The method comes from brain imaging. There, variance partitioning fits a prediction model on each set of explanatory features alone and on all sets jointly. It then uses the differences in out-of-sample explained variance to split each brain location's signal into a part unique to each feature set and a part they share. We treat each benchmark task the way neuroscientists treat a location in the brain. Each task gets its own partition, and together the partitions form a "map" of which tasks behave like measurements of agent ability and which behave like measurements of the harness. The claim we test is concrete: tasks whose outcomes carry substantial variance unique to harness or host will be the tasks that UTBoost later found to be broken. Removing them will make agent rankings stable across evaluation setups in a way that simple flakiness filtering cannot.
Core Claim
Fix a set of patches and re-evaluate each one under a crossed grid of evaluation conditions. A correct measuring instrument should give an outcome that depends only on the patch. Any outcome variance uniquely predicted by the parser, the test-suite version, the container build, or the host CPU is instrumentation bias, in the sense of Instrumentation bias in the use and evaluation of scientific software. For task $t$, let $R^2_P$, $R^2_E$ and $R^2_{P+E}$ be the cross-validated explained variance of its outcomes from patch features $P$ alone, evaluation features $E$ alone, and both together. The quantity we care about is
$$U_t(E) = R^2_{P+E} - R^2_P,$$
which is the predictive power the evaluation setup adds once the patch is known. On a well-behaved task $U_t(E)$ should be near zero. The shared term $R^2_P + R^2_E - R^2_{P+E}$ captures a different situation: the patch and the evaluation setup are confounded. One example is a model that only ever produces patches that slip past the regex parser.
Patch features are why this needs variance partitioning rather than a variance table. These features are high-dimensional and continuous. They include diff statistics, files touched, an embedding of the diff, and whether the patch passes a held-out augmented test set. They are also strongly collinear with which model produced the patch. Ridge regression with cross-validation, as used for collinear auditory feature spaces in The Hierarchical Cortical Organization of Human Speech Processing and Intermediate acoustic-to-semantic representations, is built for this setting. Fourier power, subjective distance, and object categories is the standard illustration that overlapping predictors need this treatment. These papers supply the method. None of them applies it outside neural or behavioral data.
Research Program
The work has two stages. In the generation stage, we run OpenHands' CodeActAgent (OpenHands) with four language models and three seeds on SWE-bench Lite (300 tasks) and Verified (500 tasks). This gives about twelve patches per task. To each task we add two probe patches. One is the empty patch, borrowing the "do-nothing" agent that scored 38% on τ-bench in the best-practices audit. The other is the gold patch.
In the measurement stage, each patch is re-evaluated under a crossed grid. Some factors cost nothing extra because they are applied to the same logs: regex parser versus queue-based parser. Others require reruns: the original versus an augmented test suite, two container rebuilds from the same recipe, two host CPU architectures, and two repeated runs. Because the measurement grid is fully crossed by design, harness and host factors are orthogonal to each other. They are not orthogonal to the patch features. The collinearity sits exactly where variance partitioning is needed.
For each task we fit cross-validated ridge or logistic-ridge models on the patch, harness, and host feature spaces and their unions, using the stacked-regression tooling of Stacked regressions and structured variance partitioning. We report the unique and shared terms with bootstrap intervals over patches.
Cross-validated partitions can come out negative for collinear predictors, a known weakness of the method. We report negative estimates rather than clipping them, and we treat a task as harness-dominated only when its bootstrap interval excludes zero. Pass/fail outcomes have a ceiling on explainable variance. We estimate that ceiling from repeated runs and normalize by it, which addresses the noise-ceiling bias noted in the variance-partitioning literature.
A second partition runs on the generation stage. Its target is the model's success on each task, and its feature spaces are model identity, seed, and host. This separates real agent stochasticity from environment effects. It extends the methodological/arbitrary/incidental taxonomy of Exploring Variation of Results from Different Experimental Conditions, which reported only per-condition spreads such as a seed-only BLEU range of 84.47 to 89.59, into unique and shared attributions.
Baselines
The naive reference point is the standard leaderboard procedure: one evaluation per patch with the default harness, and rankings computed from it. Every other method is judged by whether it changes rankings or flags tasks in ways this procedure misses.
The strongest existing method is UTBoost's harness correction. It replaces the regex parser and adds LLM-generated tests. It reports 345 false passes, 29 ranking changes, affected entries on 40.9% of the Lite and 24.4% of the Verified leaderboards, and corrected annotations in over 54% of instances. We do not compete with it as a fix. We use its corrected labels as held-out ground truth.
Our screen sees only the default and alternative configurations, never UTBoost's labels. It must rank tasks so that UTBoost-flagged instances appear near the top. We report area under the ROC curve and precision at the size of the flagged set. No reported result was found on Lacuna for any automatic method predicting UTBoost's false passes. Our comparison point is therefore the competing screens below, not a published number.
The rerun-based screen is the small-sample corrected coefficient of variation $CV^*$ from Quantified Reproducibility Assessment. It is computed per task over the same evaluation grid, alongside a raw disagreement rate across reruns. These screens get the same runs, the same tasks, and the same ranking-based scoring. A second competitor is a per-task mixed-effects ANOVA with model, seed, parser, suite, and host as categorical factors plus a model-by-parser interaction. It is fit on the same data with the same cross-validation folds.
Two ablations isolate our key ingredient. The first replaces the patch-feature space with model identity alone, which tests whether patch content matters. The second drops cross-validation and uses in-sample $R^2$. We count our method as beating the competitors if it achieves significantly higher AUROC against UTBoost labels than the $CV^*$ and ANOVA screens under a paired bootstrap over tasks. It must also raise Kendall's tau between the rankings obtained under different host and harness configurations more than removing an equal number of the most flaky tasks does.
Why not just…?
Why not just compute $CV^*$ or rerun disagreement per task? Dispersion tells you that a task's outcome varies. It does not tell you whether the variation comes from the agent's legitimate randomness or from the instrument. Many of UTBoost's false passes are not flaky at all: the regex parser passes the broken patch every time. A dispersion screen should therefore miss them, while the parser's unique variance across the parser contrast should catch them. This is a direct, falsifiable prediction. If $CV^*$ matches our AUROC, the patch-feature machinery is unnecessary and we will say so.
Why not just run a factorial ANOVA? For the orthogonal harness and host factors, ANOVA and in-sample partitioning largely agree. This is a real threat, and it is why ANOVA is a full baseline rather than a strawman. The difference lies in the patch, which is a high-dimensional collinear description rather than a few categorical levels. The "model identity only" ablation is exactly the ANOVA-style treatment of the patch. If it performs as well as rich patch features, the honest conclusion is that a mixed-effects ANOVA is the right tool.
Why not just fix the harness, as UTBoost did? Fixing removes bugs someone has already found, at roughly $1.60 per task instance in UTBoost's own accounting, and it must be redone for every new benchmark. Our screen is meant to tell auditors where to look first. If it does not retrodict UTBoost's findings on SWE-bench, it has no claim on benchmarks that have not yet been audited.
Why not just freeze the container image? Our grid includes rebuilds of the same pinned recipe and two CPU architectures. Floating-point and runtime differences persist across builds despite pinning (Provenance Design and Evolution in a Production ML Library; ARCH-COMP 2021). The host's unique variance therefore directly measures what pinning leaves behind. If it is near zero everywhere, that is itself a useful negative result.
Decisive Checks
The proposal succeeds if three things hold. First, the empty-patch probe passes predominantly on tasks with high unique evaluation variance. Second, the per-task harness term predicts UTBoost's false passes better than $CV^*$, rerun disagreement, and ANOVA on SWE-bench Lite and Verified. Third, re-ranking agents on tasks with low unique evaluation variance makes rankings stable across parser and host configurations. It fails, and should be abandoned in favor of simpler dispersion screens, if any of these does not hold.
References
- UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- Instrumentation bias in the use and evaluation of scientific software: recommendations for reproducible practices in the computational sciences
- The Hierarchical Cortical Organization of Human Speech Processing
- Intermediate acoustic-to-semantic representations link behavioral and neural responses to natural sounds
- Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents
- Stacked regressions and structured variance partitioning for interpretable brain maps
- Exploring Variation of Results from Different Experimental Conditions
- Quantified Reproducibility Assessment of NLP Results
- Provenance Design and Evolution in a Production ML Library
- ARCH-COMP 2021 Category Report: Falsification with Validation of Results