Accelerate Bench: Evaluating Agents for Application-Level Program Optimization

Bowen Liu, Yuhan Li, Nuo Chen, Weile Luo, Miao Peng, Qintong Zhang, Qifan Zhang, Jia Li, Ziqi Gao, Lydell HuangarXiv 2026 · arXiv

TL;DR

The authors introduce Accelerate Bench, a new benchmark designed to evaluate AI coding agents on end-to-end, application-level program optimization. Unlike previous benchmarks that focus on localized code fixes, Accelerate Bench requires agents to diagnose performance bottlenecks and coordinate complex changes across algorithms, build configurations, and runtime settings to achieve real-world speedups.

What Problem Does This Paper Address?

Software performance engineering is a notoriously difficult, iterative process. It’s not just about writing "better" code; it’s about understanding how a complete application interacts with the entire software stack—from compiler flags and data structures to hardware-level parallelism.

Existing AI benchmarks like SWE-bench or GSO (Gold Standard Optimization) typically evaluate agents on their ability to reproduce specific human-authored commits or fix isolated bugs. This misses the "discovery" phase of optimization: an agent must first figure out where the time is being spent before deciding how to intervene. Furthermore, optimizations often interact; a change in one layer of the stack might be useless—or even detrimental—without a corresponding change elsewhere. Accelerate Bench fills this gap by providing a "workload-defined" evaluation where the goal is simply to make a specific task run faster while remaining correct.

Comparison of commit-derived tasks vs. workload-defined tasks in Accelerate Bench.

Methods

Accelerate Bench consists of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions (e.g., fluid dynamics, molecular simulation, quantum circuits).

The evaluation protocol mimics a real engineering workflow:

  1. Baseline & Instructions: Agents are given a working initial program, a description of the workload, and clear correctness requirements.
  2. Iterative Development: Agents have an 8-hour window to inspect, profile, edit, and build the program. They can use a "Public OJ" (Online Judge) to get feedback on correctness and runtime for small public inputs.
  3. Private Verification: The final submission is scored by a separate, hidden verifier using larger, randomized private inputs to ensure the optimizations generalize and aren't just "overfitting" to the test cases.
  4. Scoring: The primary metric is Score@1, which represents the mean fraction of baseline runtime saved across all tasks, gated by a strict correctness check.

Datasets

The paper introduces Accelerate Bench v1.1, featuring 32 tasks across eight application areas: Molecular Simulation, Fluids, Transport, Bioinfo, Numerics, Quantum, ML, and Scientific I/O. These tasks are curated from prestigious competitions like the ASC Student Supercomputer Challenge and the SC/ISC Student Cluster Competitions.

Key Insights and Ideas

  • Coordination is King: The authors found that identical source code can vary in performance by nearly an order of magnitude (0.29x to 2.12x) based solely on build and runtime settings. This highlights that agents cannot ignore the "environment" layer of optimization.
  • Non-Linear Gains: Code changes often exhibit strong dependencies. In one QiboTN task, a specific code change provided a negligible 1.01x speedup on its own but jumped to 6.4x when combined with another architectural change.
  • Correctness is Easy, Speed is Hard: Most evaluated agents were highly successful at producing correct programs, but the magnitude of their speedups varied wildly. The "size" of the speedup is what truly separates the top-tier agents from the rest.
  • High-Leverage Techniques: Parallelism and vectorization were identified as the most effective techniques for reaching target speedups, though they are also the most complex for agents to implement correctly.
Optimization techniques used by agents and their impact on performance.

Key Results

The study evaluated 16 coding agents, pairing various LLMs (like Claude 3.5, GPT-4, DeepSeek) with different tool-using harnesses.

  • Claude Leads the Pack: Claude Sonnet 5.5 and Claude Opus 5.5 were the clear leaders, achieving Score@1 values of 67.4% and 59.4%, respectively.
  • Speedup Distribution: The top two agents achieved at least a 2x speedup on 22 or more of the 32 tasks. In contrast, the other fourteen agents only reached that mark on 8 to 17 tasks.
  • Performance by Domain: Agents generally performed best in areas like Quantum and ML, while Bioinfo and Transport proved more challenging.
Score@1 performance across different application areas for the top agents.
Distribution of outcomes (correctness and speedup) for the evaluated agents.

Limitations and Future Work

  • CPU Focus: The current benchmark is limited to CPU-based workloads. Extending this to GPU-accelerated tasks (e.g., CUDA, ROCm) is a natural and necessary next step given the importance of GPUs in modern high-performance computing.
  • Baseline Sensitivity: Speedups are relative to the provided baselines. While the authors tried to provide "reasonable" baselines, the starting point of tuning can influence the perceived gain.
  • Resource Constraints: Tasks are scaled to run on a single node within practical time limits. Performance characteristics might shift when moving to massive, multi-node distributed systems.
  • Agent Trajectories: While the paper analyzes how agents optimize, more work is needed to understand the "reasoning" failures that prevent agents from discovering even better optimizations.

Summary updated

Abstract

Application optimization requires coding agents to discover useful changes and coordinate them across a complete program while preserving the required computation. We introduce Accelerate Bench, a benchmark of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions. Each task provides a working program and specifies the workload, correctness requirements, timing boundaries, and permitted edits, leaving agents to choose where and how to optimize. Public testing supports iterative development, and a separate verifier scores the final submission. On benchmark version 1.1, we evaluate 16 coding agents, each pairing a model with a harness. Their Score@1 ranges from 25.5% to 67.4%. Most of their programs are correct, and the size of the speedup separates them: the two leading agents are at least 2x faster on 22 or more of the 32 tasks, while the other fourteen reach 2x on 8 to 17 and leave 125 of their 412 correct outcomes below 1.25x. Controlled replays show why coordination across an application matters: identical source code runs between 0.29x and 2.12x depending only on build and runtime settings, and one code change is worth 1.01x or 6.4x depending on another. Accelerate Bench pairs executable workloads with agents' programs and trajectories to evaluate both their gains and the process behind them.
BibTeX
@misc{liu2026accelerate,
  author = {Bowen Liu and Yuhan Li and Nuo Chen and Weile Luo and Miao Peng and Qintong Zhang and Qifan Zhang and Jia Li and Ziqi Gao and Lydell Huang},
  title = {{Accelerate Bench: Evaluating Agents for Application-Level Program Optimization}},
  year = {2026},
  eprint = {2611.20882},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20882},
}
Figure