Agentic GPU Kernel Optimization

This direction is split into other directions: hardware-specific optimization complexity. Its last version stays readable and citable here.

Version 1 of 1140 papers145 conceptsfrozen 2026-10-04

Writing high-performance GPU kernels is often described as a "dark art." While a developer can write functionally correct CUDA code in minutes, transforming that code into something that approaches the theoretical peak performance of the hardware requires navigating a labyrinth of memory hierarchies, thread synchronization, and instruction-level parallelism.

Traditionally, this optimization process is manual and iterative: an expert engineer writes a kernel, profiles it using tools like NVIDIA Nsight, identifies bottlenecks (e.g., uncoalesced memory access or low occupancy), and rewrites the code. Recent research has shifted this burden from human experts to agentic LLM frameworks, which automate this "profile-and-refine" loop to discover optimizations that even expert compilers often miss.

The Performance Gap in GPU Programming

The fundamental challenge in GPU programming is the disconnect between high-level mathematical logic and low-level hardware constraints. As noted by Anne Ouyang and Simon Guo in KernelBench, even frontier models like OpenAI o1 struggle to write efficient kernels in a single shot because peak performance requires deep reasoning about execution patterns, not just syntactic correctness.

Standard compilers are often too conservative to exploit hardware-specific "tricks." This creates a performance gap where hand-tuned kernels can be orders of magnitude faster than generic implementations. Frameworks like ThunderKittens by Benjamin Frederick Spector attempt to bridge this with better abstractions, but the ultimate goal is to automate the discovery of these optimizations entirely.

The Agentic Optimization Loop

The core idea behind recent breakthroughs like CudaForge and Astra is to treat kernel optimization as a multi-agent search problem. Instead of a single prompt, these systems deploy a "team" of specialized agents:

  1. The Coder: Generates an initial kernel or a set of mutations.
  2. The Profiler/Judge: Executes the code on real hardware and extracts performance metrics (e.g., throughput, memory bandwidth utilization).
  3. The Planner: Analyzes the feedback to suggest specific architectural changes, such as register blocking or loop unrolling.

Zijian Zhang and colleagues demonstrate in CudaForge that by providing LLMs with real-time hardware feedback from Nsight Compute, the agents can achieve a 1.68× speedup over PyTorch baselines without any specialized training. This "hardware-in-the-loop" approach allows the model to "see" the physical bottlenecks of its code.

Decoupling Strategy from Implementation

A recurring theme in the most successful frameworks is the separation of high-level optimization strategy from low-level code generation. Jiaming Guo and Yunji Chen formalize this as the Macro-Thinking Micro-Coding (MTMC) paradigm in QiMeng-Kernel.

In this framework, the "Macro" agent decides on a global strategy—such as how to tile the data or which memory hierarchy to use—while the "Micro" agent handles the tedious task of writing the specific CUDA instructions. This prevents the LLM from getting lost in the "lexical complexity" of CUDA, a problem highlighted by Yu-Sheng Lin in earlier work on UMI operators. By decoupling these concerns, QiMeng-Kernel achieves up to a 34× speedup over standard LLM-generated code.

Search and Evolutionary Refinement

Because the search space for optimizations is vast, many researchers are moving beyond simple linear chains of thought toward tree-based and evolutionary searches.

  • Strategic Search: Juncheng Dong and the STARK team use a tree-based search to explore multiple optimization paths simultaneously, allowing the system to backtrack if a particular strategy (like aggressive loop unrolling) leads to register spilling.
  • Evolutionary Evolution: EvoEngineer by Ping Guo applies evolutionary algorithms to "mutate" kernel code, keeping only the fastest versions for the next generation.
  • Reinforcement Learning: CUDA-L1 by Albert Wang uses contrastive reinforcement learning to train models specifically on the difference between slow and fast kernels, helping the model internalize the "intuition" of an expert human optimizer.

Beyond CUDA: Multi-Platform Portability

While much of the focus is on NVIDIA's CUDA, the agentic approach is particularly valuable for "niche" or emerging hardware where documentation is sparse and expert programmers are rare. Yinghui Zhang introduced MultiKernelBench to evaluate LLMs across platforms like Huawei's AscendC and Google's TPUs. Similarly, GEAK from AMD focuses on automating Triton kernels for AMD GPUs, demonstrating that agentic loops can bridge the "software gap" that often hinders the adoption of new AI accelerators.

By automating the iterative profiling and refinement process, these agentic systems are turning the "dark art" of kernel optimization into a scalable, automated engineering discipline. As Carlo Baronio notes in Kevin, the goal is to replicate the expert's workflow: "write, profile, tweak, and repeat"—but at the speed of an LLM.

Papers in this version

Showing 60 of 140 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_agentic_gpu_kernel_optimization_7122fe89,
  title = {Agentic GPU Kernel Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/agentic-gpu-kernel-optimization/rdr_7122fe89b3a141c79e72c97fdd7726e1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_agentic_gpu_kernel_optimization_7122fe89_v1,
  title = {Agentic GPU Kernel Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/agentic-gpu-kernel-optimization/rdr_7122fe89b3a141c79e72c97fdd7726e1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}