Abstract
Writing high-performance CUDA kernels is a complex task requiring deep expertise in hardware-software co-design. While Large Language Models (LLMs) have shown promise in code generation, they often struggle with the nuanced optimizations required for peak GPU performance. We present cuPilot, a multi-agent framework that treats kernel optimization as a strategy-coordinated evolutionary process. By decoupling high-level optimization strategies from low-level implementation, cuPilot achieves an average 3.09x speedup over PyTorch across 100 diverse kernels. Our approach introduces a "Strategy" abstraction layer and hardware-aware "Roofline-Guided Prompting" to overcome the limitations of conventional automated code generation.
Introduction
High-performance CUDA programming is often considered a "black art." Developers must balance thread-level parallelism, memory hierarchy utilization, and hardware-specific features like Tensor Cores. Existing automated frameworks for kernel generation frequently encounter three primary "mismatches":
- Mismatched Crossover: LLMs struggle to combine optimizations from two different kernels because the logic is often deeply interleaved with complex syntax.
- Mismatched Fitness: Generic feedback (e.g., "the kernel is slow") lacks the diagnostic depth needed for an LLM to make targeted improvements.
- Mismatched Initialization: Starting from a sparse or random population often leads to local optima rather than peak performance.
cuPilot addresses these challenges by introducing an intermediate "Strategy" layer, allowing the system to reason about optimizations (the what) independently of the code (the how).
The cuPilot Framework
Strategy-Coordinated Evolution (SCE)
At the heart of cuPilot is the Strategy-Coordinated Evolution (SCE) mechanism. Instead of performing crossover on raw CUDA code—which often leads to syntax or functional errors—cuPilot evolves a population of high-level Strategies (e.g., "Tiling," "Shared Memory Swizzling," or "Double Buffering"). A specialized Strategy Translator agent then applies these combined strategies to the kernel code. This abstraction ensures that the underlying optimization logic is preserved even as the code structure changes.
Roofline-Guided Prompting
To provide actionable feedback, cuPilot utilizes a Roofline Prophet agent. By positioning a kernel's performance on the GPU's Roofline Model, the system identifies whether a kernel is compute-bound, memory-bound, or in a transition zone. This hardware-aware diagnostic allows the framework to provide specific, high-utility advice. For instance, if a kernel is memory-bound, the system prompts for memory-specific strategies like padding or swizzling rather than wasting computational resources on compute-centric optimizations.
Multi-Agent Orchestration
The framework employs a suite of specialized agents to manage the evolutionary pipeline:
- SCE Manager: Oversees the population and selection process.
- Strategy Translator: Maps high-level strategies to concrete CUDA implementations.
- Kernel Revisor: Uses compiler and profiler feedback (NVCC, NCU) to automatically fix syntax and functional errors.
Evaluation and Results
Performance Benchmarks
We evaluated cuPilot using KernelBench, a suite of 100 level-1 kernels, and a specialized GEMM case study. cuPilot achieved an average speedup of 3.09x over PyTorch baselines. In complex GEMM tasks, the framework reached a 4.06x speedup, successfully implementing advanced techniques such as Tensor Core invocation and asynchronous memory copying—optimizations that typically require significant manual effort from human experts.
Ablation Studies
Ablation experiments highlight the necessity of the framework's core components. Removing Roofline-Guided Prompting significantly slowed convergence and resulted in lower final performance across all tested kernel types (GEMM, GEMV, Conv-3D, and Softmax). Similarly, the use of a Strategy Pool for population initialization via Retrieval-Augmented Generation (RAG) was found to be vital for avoiding local optima.
Discussion and Conclusion
cuPilot demonstrates that the key to effective LLM-driven code optimization lies in providing the model with high-level abstractions and hardware-specific context. By treating optimization as a coordinated strategy evolution rather than a simple code-writing task, cuPilot bridges the gap between automated generation and expert-level performance.
Future work will focus on scaling the framework to handle multi-operator "level-2" and "level-3" kernels and extending support to diverse hardware architectures beyond NVIDIA A100 GPUs. The generated high-performance kernels are available for the community at https://github.com/champloo2878/cuPilot-Kernels.git.
Summary updated