Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher RéarXiv 2026 · arXiv

TL;DR

Agentic benchmarks are not static datasets but complex software systems that "rot" over time as models evolve and infrastructure shifts. This paper introduces a continuous-validation workflow that identifies and repairs these defects, showing that fixing "benchmark maintenance debt" can shift model accuracy by over 10 percentage points and reveal that many performance gaps are simply artifacts of restrictive time budgets.

What Problem Does This Paper Address?

In the rapidly moving world of Large Language Models (LLMs), benchmarks like Terminal-Bench are used to evaluate how well agents can perform complex coding and terminal tasks. However, these benchmarks suffer from benchmark maintenance debt.

As models get smarter, they find valid solution paths that the original benchmark designers didn't anticipate, leading to "false failures" where a correct answer is rejected by an outdated verifier. Simultaneously, external dependencies (like APIs or software packages) change, and infrastructure differences between evaluation environments can cause tasks to break for reasons unrelated to the agent's actual capability. The authors argue that benchmarks should be treated like software: they require versioning, bug tracking, and continuous maintenance to remain valid.

Figure 1: Failure surfaces along the agentic-task lifecycle. From authoring to execution and verification, each stage can admit a failure that contributes to benchmark maintenance debt. Colors mark the three failure categories (blue: instruction–verification mismatch, orange: environment issues, red: task drift).

Methods

The authors propose a continuous-validation workflow designed to automate the "triage and repair" of benchmark tasks. The pipeline consists of four main stages:

  1. Collect Evidence: An agent gathers data from static task files, prior execution traces, and community bug reports.
  2. Audit Benchmark: An "auditor agent" (a high-capability LLM) applies a rubric to identify defects like instruction-verification mismatches or environment issues.
  3. Verify Repair: A "verifier agent" attempts to run the known-correct "oracle" solution against the proposed fix to ensure the task is actually solvable and the fix works.
  4. Release Update: A human maintainer reviews the verified repairs before shipping a new version of the benchmark.
Figure 2: Agentic pipeline for continuous benchmark validation. An evidence collector agent gathers complementary evidence streams: static task files, prior agent traces, and external reports. An auditor agent applies audit principles for specification fidelity, tests, resources, and risk, then drafts a cited repair, which is checked against evidence while /oracle rebuilds the task and runs the reference solution. The repair workflow can open a draft PR with evidence; maintainer review gates the verified release.

Datasets

The primary focus is on Terminal-Bench, a benchmark for terminal-based coding agents.

  • Terminal-Bench 2.0: The baseline version, which already had extensive human review but still accumulated defects.
  • Terminal-Bench 2.1: The updated version released by the authors, containing repairs for 28 tasks (roughly 30% of the benchmark).
  • Harbor-Index v0.1: Used as an external validation set to prove the workflow transfers to other agentic benchmarks.

Key Insights and Ideas

  • Benchmark Rot is Pervasive: Even a "well-validated" benchmark like Terminal-Bench 2.0 had issues in 30% of its tasks within months of release. The most common issue (57% of defects) was a mismatch between what the instructions asked for and what the verifier actually checked.
  • Resource Budgets are Part of the Spec: A model's "accuracy" is inextricably linked to its resource budget (time, memory, tokens). If a benchmark has a tight time limit, it might measure speed rather than reasoning capability.
  • The "Harness Gap" is Often a Time Gap: Different agent "harnesses" (the interface between the model and the terminal) often show large performance differences. The authors found that simply extending the time limit closed most of these gaps, suggesting the "better" harness was often just faster, not more capable.

Key Results

  • Significant Accuracy Shifts: Repairing the benchmark tasks in Terminal-Bench 2.1 led to an average accuracy increase of over 10 percentage points across 19 different agents.
  • Leaderboard Volatility: While overall rankings remained highly correlated (Spearman $\rho = 0.958$), individual models moved by up to three ranks. This suggests that absolute performance claims on older benchmark versions may be misleading.
Comparison of accuracy between Terminal-Bench 2.0 and 2.1, showing uneven gains across different agents.
  • Closing the Gap: For GPT-5.4, extending the wall-clock time limit closed a 20 percentage point gap between different harnesses, proving that resource allocation is a critical, often overlooked variable in evaluation.
Extending the time budget closes the performance gap between different agent harnesses.

Limitations and Future Work

While the workflow is highly effective at triaging issues, it is not yet fully autonomous. The authors achieved 57.1% precision and 71.4% recall compared to human maintainers, meaning human-in-the-loop review remains essential.

Future work includes developing more robust "trajectory-level" judges that can look at an agent's step-by-step reasoning rather than just the final state, and further exploring how to set "principled floors" for resource budgets to ensure benchmarks measure capability rather than just execution speed. The authors conclude that benchmarks must move toward a model of continuous versioning—much like the software they are meant to evaluate.

Summary updated

Abstract

Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to evaluation infrastructure can invalidate assumptions embedded in tasks and tests. We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. We apply this workflow to Terminal-Bench 2.0, a benchmark that underwent extensive human and LM-assisted review and red-teaming before release. Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958). We evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock can materially change agent performance: For example, extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp. Benchmark maintenance is therefore not only about repairing task artifacts: resource budgets are part of the measurement specification and must be validated alongside the tasks themselves. More broadly, benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.
BibTeX
@misc{buchanan2026benchmarks,
  author = {E. Kelly Buchanan and Crystal Zhou and Scott W. Linderman and Christopher Ré},
  title = {{Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks}},
  year = {2026},
  eprint = {2611.08417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08417},
}
Figure