TL;DR
CompoundingKernel is an autonomous agent system that writes high-performance GPU kernels for NVIDIA’s Blackwell (B200) architecture starting from nothing but a PyTorch definition. By using an LLM-driven "compounding" search that builds on an archive of previous successes, the system achieved #1 rankings on the SOL-ExecBench leaderboard, outperforming standard baselines by an average of 4.4x.
What Problem Does This Paper Address?
In the world of high-performance computing, hardware often evolves faster than the software libraries designed to run on it. When NVIDIA releases a new architecture like Blackwell, there is a significant lag before standard libraries (like cuBLAS or Triton) are fully optimized to exploit new features like asynchronous tensor-core operations (tcgen05) or the Tensor Memory Accelerator (TMA).
Writing these kernels manually is a grueling process of "design, measure, fix, repeat." While LLMs are increasingly good at writing code, they struggle with the deep architectural search required to find the "speed-of-light" (SOL) performance limit. Previous agentic approaches often "hallucinate" performance or fall into "reward hacking"—finding ways to trick a timer rather than actually making the code faster. CompoundingKernel aims to automate this entire optimization loop while maintaining rigorous correctness.
Methods
The authors introduce a system built on four nested loops, moving from fast, automated iterations to slower, human-in-the-loop oversight:
- The Child Loop (Fastest): A child agent (using Claude 3.5 Opus) is given a parent kernel and a specific direction. It edits, compiles, and tests the code on a physical B200 GPU. It iterates until the code is correct on at least one workload.
- The Controller Loop: This is the "brain" of the search. An LLM controller queries an external Archive—a database of every correct kernel found so far and its lineage. It decides which kernel to refine next and what strategy to try, ensuring the search doesn't just hill-climb but explores new "roots" or designs.
- The Team Loop: Lead agents monitor the fleet, fixing the "harness" (the prompts and tools) via warm restarts. A warm restart allows the system to update its logic without losing the progress stored in the Archive.
- The People Loop (Slowest): Humans use a dashboard to monitor progress and decide which kernels are robust enough to submit to public leaderboards.
Crucially, the system uses external verification. Correctness and timing are measured outside the LLM's context to ensure the agents are learning from ground-truth hardware performance, not their own predictions.
Datasets
The system was evaluated using SOL-ExecBench, NVIDIA’s public benchmark for AI-generated kernels. It consists of 235 problems derived from 124 real-world AI models, categorized into:
- L1: Single operations (94 problems).
- L2: Fused operations (82 problems).
- Quantization: Low-precision operations (33 problems).
- FlashInfer-Bench: Specialized kernels for inference (26 problems).
Key Insights and Ideas
- Slower Kernels as Stepping Stones: One of the most surprising findings was that 32% of final best kernels descended from a parent that was at least 10% slower than the best kernel available at that time. This suggests that "worse" code can sometimes provide a better architectural foundation for future breakthroughs—a concept the authors call "compounding."
- Throughput Beats "Smart" Picks: When compared to a traditional coded UCB (Upper Confidence Bound) tree search, the LLM controller won on 35 of 38 problems. Interestingly, it wasn't necessarily better at picking the best node; it won because it was better at keeping the parallel agents busy, making 2.4x more correct attempts in the same wall-clock time.
- The Measurement Bottleneck: The authors found that scaling the system isn't just about adding more LLM agents. The real limit is the rate of reliable hardware measurements. Because high-fidelity timing requires locking the GPU clock and exclusive access, the physical hardware becomes the primary bottleneck.
- Zero-Knowledge Start: The agents start "from zero," meaning they see the PyTorch math but never see reference CUDA kernels or official leaderboard scores. They learn entirely from their own local timing and correctness checks.
Key Results
- #1 Rankings: As of November 2026, CompoundingKernel holds the #1 spot on the L1, L2, and Quantization boards of SOL-ExecBench.
- Massive Speedups: The generated kernels are 4.40x faster (geometric mean) than the official PyTorch baselines. Locally, they are 11.2x faster than
torch.compile. - High Reliability: Despite the complexity of Blackwell-native code (often exceeding 7,000 lines per kernel), 98.4% of the candidates evaluated in the final run were correct.
- Extreme Outliers: In some cases, the speedup was astronomical, with one kernel (L1/094) running 1062x faster than the baseline.
Limitations and Future Work
- Hardware Lock-in: The current system is highly optimized for NVIDIA B200. It remains to be seen how easily the "harness" transfers to other architectures like AMD’s Instinct or custom AI accelerators.
- Verification Costs: The system requires significant "test-time compute" on expensive B200 GPUs. The authors suggest that developing "cheaper verification" methods (perhaps using simulators or partial runs) is a priority for future scaling.
- Formal Correctness: While the kernels pass exhaustive test suites, they are not formally proven. Current formal verification tools struggle to model the asynchronous nature of Blackwell’s latest instructions.
- End-to-End Integration: The next step is integrating these kernels into full serving frameworks to measure real-world impact on metrics like "Time to First Token" (TTFT) in Large Language Models.
Summary updated