cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution

Jinwu Chen, Qidie Wu, Bin Li, Lin Ma, Xin Si, Yang Hu, Shouyi Yin, Jun YangarXiv 2025 · DOI · arXiv

Abstract

Writing high-performance CUDA kernels is a complex task requiring deep expertise in hardware-software co-design. While Large Language Models (LLMs) have shown promise in code generation, they often struggle with the nuanced optimizations required for peak GPU performance. We present cuPilot, a multi-agent framework that treats kernel optimization as a strategy-coordinated evolutionary process. By decoupling high-level optimization strategies from low-level implementation, cuPilot achieves an average 3.09x speedup over PyTorch across 100 diverse kernels. Our approach introduces a "Strategy" abstraction layer and hardware-aware "Roofline-Guided Prompting" to overcome the limitations of conventional automated code generation.

Introduction

High-performance CUDA programming is often considered a "black art." Developers must balance thread-level parallelism, memory hierarchy utilization, and hardware-specific features like Tensor Cores. Existing automated frameworks for kernel generation frequently encounter three primary "mismatches":

  1. Mismatched Crossover: LLMs struggle to combine optimizations from two different kernels because the logic is often deeply interleaved with complex syntax.
  2. Mismatched Fitness: Generic feedback (e.g., "the kernel is slow") lacks the diagnostic depth needed for an LLM to make targeted improvements.
  3. Mismatched Initialization: Starting from a sparse or random population often leads to local optima rather than peak performance.

cuPilot addresses these challenges by introducing an intermediate "Strategy" layer, allowing the system to reason about optimizations (the what) independently of the code (the how).

The cuPilot Framework

Strategy-Coordinated Evolution (SCE)

At the heart of cuPilot is the Strategy-Coordinated Evolution (SCE) mechanism. Instead of performing crossover on raw CUDA code—which often leads to syntax or functional errors—cuPilot evolves a population of high-level Strategies (e.g., "Tiling," "Shared Memory Swizzling," or "Double Buffering"). A specialized Strategy Translator agent then applies these combined strategies to the kernel code. This abstraction ensures that the underlying optimization logic is preserved even as the code structure changes.

The cuPilot multi-agent architecture and Strategy-Coordinated Evolution (SCE) workflow.

Roofline-Guided Prompting

To provide actionable feedback, cuPilot utilizes a Roofline Prophet agent. By positioning a kernel's performance on the GPU's Roofline Model, the system identifies whether a kernel is compute-bound, memory-bound, or in a transition zone. This hardware-aware diagnostic allows the framework to provide specific, high-utility advice. For instance, if a kernel is memory-bound, the system prompts for memory-specific strategies like padding or swizzling rather than wasting computational resources on compute-centric optimizations.

Multi-Agent Orchestration

The framework employs a suite of specialized agents to manage the evolutionary pipeline:

  • SCE Manager: Oversees the population and selection process.
  • Strategy Translator: Maps high-level strategies to concrete CUDA implementations.
  • Kernel Revisor: Uses compiler and profiler feedback (NVCC, NCU) to automatically fix syntax and functional errors.

Evaluation and Results

Performance Benchmarks

We evaluated cuPilot using KernelBench, a suite of 100 level-1 kernels, and a specialized GEMM case study. cuPilot achieved an average speedup of 3.09x over PyTorch baselines. In complex GEMM tasks, the framework reached a 4.06x speedup, successfully implementing advanced techniques such as Tensor Core invocation and asynchronous memory copying—optimizations that typically require significant manual effort from human experts.

Performance comparison of cuPilot with and without Roofline-Guided Prompting across various kernel types, showing faster convergence and lower latency.

Ablation Studies

Ablation experiments highlight the necessity of the framework's core components. Removing Roofline-Guided Prompting significantly slowed convergence and resulted in lower final performance across all tested kernel types (GEMM, GEMV, Conv-3D, and Softmax). Similarly, the use of a Strategy Pool for population initialization via Retrieval-Augmented Generation (RAG) was found to be vital for avoiding local optima.

Discussion and Conclusion

cuPilot demonstrates that the key to effective LLM-driven code optimization lies in providing the model with high-level abstractions and hardware-specific context. By treating optimization as a coordinated strategy evolution rather than a simple code-writing task, cuPilot bridges the gap between automated generation and expert-level performance.

Future work will focus on scaling the framework to handle multi-operator "level-2" and "level-3" kernels and extending support to diverse hardware architectures beyond NVIDIA A100 GPUs. The generated high-performance kernels are available for the community at https://github.com/champloo2878/cuPilot-Kernels.git.

Summary updated

Abstract

Optimizing CUDA kernels is a challenging and labor-intensive task, given the need for hardware-software co-design expertise and the proprietary nature of high-performance kernel libraries. While recent large language models (LLMs) combined with evolutionary algorithms show promise in automatic kernel optimization, existing approaches often fall short in performance due to their suboptimal agent designs and mismatched evolution representations. This work identifies these mismatches and proposes cuPilot, a strategy-coordinated multi-agent framework that introduces strategy as an intermediate semantic representation for kernel evolution. Key contributions include a strategy-coordinated evolution algorithm, roofline-guided prompting, and strategy-level population initialization. Experimental results show that the generated kernels by cuPilot achieve an average speed up of 3.09$\times$ over PyTorch on a benchmark of 100 kernels. On the GEMM tasks, cuPilot showcases sophisticated optimizations and achieves high utilization of critical hardware units. The generated kernels are open-sourced at https://github.com/champloo2878/cuPilot-Kernels.git.
BibTeX
@misc{chen2025cupilot,
  author = {Jinwu Chen and Qidie Wu and Bin Li and Lin Ma and Xin Si and Yang Hu and Shouyi Yin and Jun Yang},
  title = {{cuPilot: A Strategy-Coordinated Multi-agent Framework for CUDA Kernel Evolution}},
  year = {2025},
  eprint = {2512.16465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.16465},
}
Figure