TL;DR
AscendGYM is a new benchmark designed to evaluate how effectively AI agents can optimize existing high-performance kernels for Ascend NPUs using real-world LLM workloads. It introduces the Speedup–Token Score (STS), a novel metric that balances performance gains against the computational cost (tokens) required to achieve them, ensuring that optimizations are both effective and economically viable.
What Problem Does This Paper Address?
Efficient Large Language Model (LLM) training and inference rely heavily on highly optimized "kernels"—the fundamental mathematical operations (like matrix multiplication or normalization) that run on specialized hardware like Ascend NPUs.
While several benchmarks exist for generating kernels from scratch, they often fall short in three areas:
- Optimization vs. Generation: In the real world, engineers usually start with a working kernel and try to make it faster. Most benchmarks don't test this iterative optimization process.
- Synthetic vs. Real Workloads: Performance varies wildly based on tensor shapes and data types. Many benchmarks use synthetic configurations that don't reflect the actual bottlenecks found in production LLM pipelines.
- The Hidden Cost of Search: Finding a faster kernel can be incredibly expensive in terms of LLM tokens. A 10% speedup that costs $1,000 in API fees might not be worth it. Existing metrics ignore this trade-off.
Methods
The authors developed AscendGYM, a framework that provides a realistic "gym" for AI agents to practice kernel optimization.
The AscendForge Sandbox
To make evaluation reliable, the authors built AscendForge, an execution sandbox that handles the complex plumbing of NPU development. It automates compilation, correctness checking (comparing outputs against a "golden" CPU reference), and precise performance profiling.
The Speedup–Token Score (STS)
The standout contribution is the STS metric. It rewards the final speedup achieved but applies a penalty for "delayed improvements." If an agent finds a massive speedup in its first few attempts, it scores higher than an agent that finds the same speedup after thousands of iterations. This encourages "budget-aware" optimization.
Evaluation Workflow
The workflow mimics a human engineer's loop: read the spec, run a baseline, analyze bottlenecks, propose code changes, and verify results in the sandbox.
Datasets
AscendGYM introduces a suite of 48 kernel optimization tasks categorized by difficulty:
- Batch 1 (17 kernels): Focuses on memory access patterns in basic kernels (e.g.,
RMSNorm,SwiGLU). - Batch 2 (17 kernels): Covers multi-step computations like quantization.
- Batch 3 (14 kernels): Targets complex attention mechanisms and fused operators.
Crucially, the tensor shapes and data types for these kernels are extracted directly from real-world models like Qwen3-0.6B and frameworks like vLLM-Ascend, ensuring that improvements in the benchmark translate to real-world speedups.
Key Insights and Ideas
- Speedup is not enough: The authors found that two systems might reach the same final speedup but differ vastly in efficiency. One might find the gain in 5 attempts, while another takes 50. STS successfully captures this distinction.
- The "Warm Start" Advantage: Using a "warm" configuration—where the agent is provided with "lessons" or strategies learned from optimizing other kernels—significantly improves the STS. This suggests that cross-task experience is vital for efficient optimization.
- Correctness $\neq$ Performance: High "Pass Rates" (producing code that works) do not guarantee high speedups. For example, GPT-5.6 passed nearly all tasks but achieved significantly lower speedups than Claude Opus 5 on certain harnesses.
- Real-world difficulty: Even for SOTA models, optimizing existing kernels is hard. Approximately 70% of evaluated model-harness combinations achieved a geometric mean speedup of less than 1.2x.
Key Results
The researchers evaluated several leading models (Claude Opus 5, GPT-5.6, GLM-5.3, Qwen series) across different agent harnesses (Claude Code, Codex, OpenCode).
- Top Performer: Claude Opus 5 paired with the Claude Code harness generally achieved the highest performance gains, though often at a higher token cost.
- Efficiency: Under a "tight" budget of 10M tokens, the "Warm" default harness configuration outperformed "Cold" starts significantly, proving that accumulated optimization strategies are a force multiplier.
- Task Coverage: While some models could accelerate all tasks in Batch 1, only a few could achieve speedups greater than 2x on more than a handful of kernels (e.g.,
fused_gdn_gatingsaw a 3.29x improvement).
Limitations and Future Work
- Scope: This initial report focuses primarily on the 17 kernels in Batch 1. Results for the more complex Batch 2 and 3 kernels are pending.
- Hardware Specificity: The benchmark is currently tailored to Ascend 910C NPUs. While the methodology is generalizable, the specific kernels and optimizations are hardware-dependent.
- Future Directions: The authors plan to expand the task suite to include more diverse workloads and explore graph-level optimizations (fusing multiple kernels together) rather than just single-kernel tuning.
Summary updated