Large Language Models (LLMs) have become remarkably proficient at generating code that is functionally correct. However, in the domain of High-Performance Computing (HPC) and GPU kernel development, "correct" is only the baseline. The real challenge is performance: writing code that effectively utilizes hardware-specific features like cache hierarchies, register pressure, and memory access patterns.
Standard LLMs lack "hardware intuition." They are trained on massive corpora of source code but rarely on the execution metrics of that code. Consequently, they often produce code that compiles but fails to achieve near-peak hardware performance. A new research direction addresses this by closing the loop between the LLM and the physical hardware, transforming code generation from a one-shot prediction into an iterative, profiling-guided optimization process.
The Performance Gap and the Need for Feedback
The primary bottleneck in automated performance engineering is that LLMs lack visibility into the execution environment. As noted by Keita Teranishi and colleagues in their roadmap for HPC software, general-purpose models often fail to account for the performance implications of their suggestions, leading to significant inefficiencies in large-scale simulations.
To bridge this gap, researchers are integrating hardware profilers directly into the LLM's reasoning loop. In frameworks like CudaForge, developed by Zijian Zhang and others, a "Coder" agent generates a kernel, which is then executed and analyzed by a "Judge" agent using real-time metrics from tools like NVIDIA Nsight Compute. This feedback—detailing warp occupancy, memory stalls, and throughput—allows the model to identify and resolve low-level bottlenecks that are invisible at the source-code level. Similarly, TritonForge, introduced by Haonan Li, uses this iterative loop to optimize Triton kernels, achieving significant speedups by reacting to real-world profiling data.
Decoupling Strategy from Implementation
Optimizing a GPU kernel requires two distinct types of thinking: high-level algorithmic strategy (e.g., tiling, swizzling, or parallelization schemes) and low-level code implementation (e.g., pointer arithmetic or register management).
The MTMC (Macro-Thinking Micro-Coding) paradigm, proposed by Jiaming Guo in QiMeng-Kernel, formalizes this by decoupling the two. The "Macro" stage determines the optimization strategy based on hardware constraints like shared memory size and SM counts, while the "Micro" stage handles the implementation. This hierarchical approach prevents the model from becoming overwhelmed by low-level details before the overall strategy is sound.
This agentic approach is also being applied to specialized hardware. Alec M. Hammond and the team behind TritorX use a similar feedback cycle to automate operator generation for Meta’s MTIA accelerator, replacing months of manual human engineering with a few hours of agentic search.
Systematic Search over Optimization Spaces
Because the space of possible optimizations is vast, researchers are moving beyond simple prompting toward structured search algorithms. Instead of asking an LLM to "make this faster," these systems treat optimization as a search problem where the LLM proposes candidates and a search algorithm evaluates them.
- Multi-Armed Bandits: Dezhi Ran and colleagues introduced KernelBand, which frames kernel optimization as a Multi-Armed Bandit problem. By clustering hardware profiling results, the system learns which optimization "arms" (strategies) are most likely to yield performance gains on specific architectures.
- Monte Carlo Tree Search (MCTS): In QiMeng-TensorOp, Xiaoqing Zhang uses MCTS to tune hardware parameters, allowing the system to outperform expert-tuned libraries like cuBLAS across different architectures.
- Evolutionary Search: The Evolution of Kernels (EoK) framework by Siyuan Chen mines "optimization ideas" from mature software libraries and uses an evolutionary process to refine RISC-V code, leveraging human-distilled insights to guide the LLM.
Building a Mental Model of Hardware
Despite these successes, LLMs still have significant "blind spots." Research by Gregory Bolet on the GPUFLOPBENCH benchmark shows that even advanced reasoning models struggle to predict the computational cost of code (like the hidden costs of division) without running it. This suggests that models lack a functional mental model of the hardware-software interface.
To address this, researchers are fine-tuning models on specialized HPC datasets. Daniel Nichols developed HPC-Coder, a model trained specifically on parallel programming patterns. Others, like Arya Tschand in SwizzlePerf, provide LLMs with explicit architectural context and profiling data to generate optimal data remapping patterns ("swizzling") that were previously the sole domain of expert human engineers.
By combining iterative feedback, hierarchical reasoning, and structured search, this research direction is turning LLMs from simple code generators into sophisticated performance engineers capable of navigating the "black art" of hardware optimization.