TL;DR
Agentic benchmarks are not static datasets but complex software systems that "rot" over time as models evolve and infrastructure shifts. This paper introduces a continuous-validation workflow that identifies and repairs these defects, showing that fixing "benchmark maintenance debt" can shift model accuracy by over 10 percentage points and reveal that many performance gaps are simply artifacts of restrictive time budgets.
What Problem Does This Paper Address?
In the rapidly moving world of Large Language Models (LLMs), benchmarks like Terminal-Bench are used to evaluate how well agents can perform complex coding and terminal tasks. However, these benchmarks suffer from benchmark maintenance debt.
As models get smarter, they find valid solution paths that the original benchmark designers didn't anticipate, leading to "false failures" where a correct answer is rejected by an outdated verifier. Simultaneously, external dependencies (like APIs or software packages) change, and infrastructure differences between evaluation environments can cause tasks to break for reasons unrelated to the agent's actual capability. The authors argue that benchmarks should be treated like software: they require versioning, bug tracking, and continuous maintenance to remain valid.
Methods
The authors propose a continuous-validation workflow designed to automate the "triage and repair" of benchmark tasks. The pipeline consists of four main stages:
- Collect Evidence: An agent gathers data from static task files, prior execution traces, and community bug reports.
- Audit Benchmark: An "auditor agent" (a high-capability LLM) applies a rubric to identify defects like instruction-verification mismatches or environment issues.
- Verify Repair: A "verifier agent" attempts to run the known-correct "oracle" solution against the proposed fix to ensure the task is actually solvable and the fix works.
- Release Update: A human maintainer reviews the verified repairs before shipping a new version of the benchmark.
Datasets
The primary focus is on Terminal-Bench, a benchmark for terminal-based coding agents.
- Terminal-Bench 2.0: The baseline version, which already had extensive human review but still accumulated defects.
- Terminal-Bench 2.1: The updated version released by the authors, containing repairs for 28 tasks (roughly 30% of the benchmark).
- Harbor-Index v0.1: Used as an external validation set to prove the workflow transfers to other agentic benchmarks.
Key Insights and Ideas
- Benchmark Rot is Pervasive: Even a "well-validated" benchmark like Terminal-Bench 2.0 had issues in 30% of its tasks within months of release. The most common issue (57% of defects) was a mismatch between what the instructions asked for and what the verifier actually checked.
- Resource Budgets are Part of the Spec: A model's "accuracy" is inextricably linked to its resource budget (time, memory, tokens). If a benchmark has a tight time limit, it might measure speed rather than reasoning capability.
- The "Harness Gap" is Often a Time Gap: Different agent "harnesses" (the interface between the model and the terminal) often show large performance differences. The authors found that simply extending the time limit closed most of these gaps, suggesting the "better" harness was often just faster, not more capable.
Key Results
- Significant Accuracy Shifts: Repairing the benchmark tasks in Terminal-Bench 2.1 led to an average accuracy increase of over 10 percentage points across 19 different agents.
- Leaderboard Volatility: While overall rankings remained highly correlated (Spearman $\rho = 0.958$), individual models moved by up to three ranks. This suggests that absolute performance claims on older benchmark versions may be misleading.
- Closing the Gap: For GPT-5.4, extending the wall-clock time limit closed a 20 percentage point gap between different harnesses, proving that resource allocation is a critical, often overlooked variable in evaluation.
Limitations and Future Work
While the workflow is highly effective at triaging issues, it is not yet fully autonomous. The authors achieved 57.1% precision and 71.4% recall compared to human maintainers, meaning human-in-the-loop review remains essential.
Future work includes developing more robust "trajectory-level" judges that can look at an agent's step-by-step reasoning rather than just the final state, and further exploring how to set "principled floors" for resource budgets to ensure benchmarks measure capability rather than just execution speed. The authors conclude that benchmarks must move toward a model of continuous versioning—much like the software they are meant to evaluate.
Summary updated