TL;DR
The authors introduce Accelerate Bench, a new benchmark designed to evaluate AI coding agents on end-to-end, application-level program optimization. Unlike previous benchmarks that focus on localized code fixes, Accelerate Bench requires agents to diagnose performance bottlenecks and coordinate complex changes across algorithms, build configurations, and runtime settings to achieve real-world speedups.
What Problem Does This Paper Address?
Software performance engineering is a notoriously difficult, iterative process. It’s not just about writing "better" code; it’s about understanding how a complete application interacts with the entire software stack—from compiler flags and data structures to hardware-level parallelism.
Existing AI benchmarks like SWE-bench or GSO (Gold Standard Optimization) typically evaluate agents on their ability to reproduce specific human-authored commits or fix isolated bugs. This misses the "discovery" phase of optimization: an agent must first figure out where the time is being spent before deciding how to intervene. Furthermore, optimizations often interact; a change in one layer of the stack might be useless—or even detrimental—without a corresponding change elsewhere. Accelerate Bench fills this gap by providing a "workload-defined" evaluation where the goal is simply to make a specific task run faster while remaining correct.
Methods
Accelerate Bench consists of 32 CPU optimization tasks adapted from scientific applications and supercomputing competitions (e.g., fluid dynamics, molecular simulation, quantum circuits).
The evaluation protocol mimics a real engineering workflow:
- Baseline & Instructions: Agents are given a working initial program, a description of the workload, and clear correctness requirements.
- Iterative Development: Agents have an 8-hour window to inspect, profile, edit, and build the program. They can use a "Public OJ" (Online Judge) to get feedback on correctness and runtime for small public inputs.
- Private Verification: The final submission is scored by a separate, hidden verifier using larger, randomized private inputs to ensure the optimizations generalize and aren't just "overfitting" to the test cases.
- Scoring: The primary metric is Score@1, which represents the mean fraction of baseline runtime saved across all tasks, gated by a strict correctness check.
Datasets
The paper introduces Accelerate Bench v1.1, featuring 32 tasks across eight application areas: Molecular Simulation, Fluids, Transport, Bioinfo, Numerics, Quantum, ML, and Scientific I/O. These tasks are curated from prestigious competitions like the ASC Student Supercomputer Challenge and the SC/ISC Student Cluster Competitions.
Key Insights and Ideas
- Coordination is King: The authors found that identical source code can vary in performance by nearly an order of magnitude (0.29x to 2.12x) based solely on build and runtime settings. This highlights that agents cannot ignore the "environment" layer of optimization.
- Non-Linear Gains: Code changes often exhibit strong dependencies. In one QiboTN task, a specific code change provided a negligible 1.01x speedup on its own but jumped to 6.4x when combined with another architectural change.
- Correctness is Easy, Speed is Hard: Most evaluated agents were highly successful at producing correct programs, but the magnitude of their speedups varied wildly. The "size" of the speedup is what truly separates the top-tier agents from the rest.
- High-Leverage Techniques: Parallelism and vectorization were identified as the most effective techniques for reaching target speedups, though they are also the most complex for agents to implement correctly.
Key Results
The study evaluated 16 coding agents, pairing various LLMs (like Claude 3.5, GPT-4, DeepSeek) with different tool-using harnesses.
- Claude Leads the Pack: Claude Sonnet 5.5 and Claude Opus 5.5 were the clear leaders, achieving Score@1 values of 67.4% and 59.4%, respectively.
- Speedup Distribution: The top two agents achieved at least a 2x speedup on 22 or more of the 32 tasks. In contrast, the other fourteen agents only reached that mark on 8 to 17 tasks.
- Performance by Domain: Agents generally performed best in areas like Quantum and ML, while Bioinfo and Transport proved more challenging.
Limitations and Future Work
- CPU Focus: The current benchmark is limited to CPU-based workloads. Extending this to GPU-accelerated tasks (e.g., CUDA, ROCm) is a natural and necessary next step given the importance of GPUs in modern high-performance computing.
- Baseline Sensitivity: Speedups are relative to the provided baselines. While the authors tried to provide "reasonable" baselines, the starting point of tuning can influence the perceived gain.
- Resource Constraints: Tasks are scaled to run on a single node within practical time limits. Performance characteristics might shift when moving to massive, multi-node distributed systems.
- Agent Trajectories: While the paper analyzes how agents optimize, more work is needed to understand the "reasoning" failures that prevent agents from discovering even better optimizations.
Summary updated