TL;DR
This paper introduces ARGUS, a novel framework that uses "data-flow invariants" to guide AI agents in optimizing GPU kernels for LLM inference. By ensuring data is choreographed correctly throughout execution, ARGUS achieves performance close to hand-optimized libraries, significantly outperforming existing AI-based approaches.
What Problem Does This Paper Address?
Modern AI, especially Large Language Models (LLMs), relies heavily on powerful GPUs for computation. While AI agents can generate functionally correct GPU kernels, their performance often lags far behind manually optimized code. This performance gap is critical because even small improvements in kernel efficiency can translate to massive cost savings in large-scale deployments. The core challenge is that achieving peak GPU performance requires a deep, coordinated understanding of complex, low-level optimizations like tiling, memory staging, and instruction scheduling. Existing AI agents struggle with this because they typically rely on sparse feedback from unit tests, making it hard for them to diagnose and fix violations of global performance constraints.
Methods
ARGUS tackles this problem with a three-pronged approach:
- Data-Flow Invariants via Tag Functions and Assertions: The core innovation is the concept of "data-flow invariants." These are compile-time specifications that describe how data should be managed throughout a kernel's execution. ARGUS uses a Pythonic Domain-Specific Language (DSL) with "tag functions" to attach symbolic annotations (like coordinates) to data elements. These tags propagate through computations. "Tag assertions" then enforce relational constraints on these tags at specific points in the code. For example, an assertion might ensure that two pieces of data expected to be paired for a matrix multiplication always carry matching tags.
- Compile-Time Static Analysis: ARGUS employs a static analysis engine that verifies these data-flow invariants entirely at compile time, incurring zero runtime overhead. It uses abstract interpretation over a "layout algebra" and SMT solvers to check these invariants. Crucially, when an invariant is violated, the compiler provides concrete "counterexamples" that pinpoint the exact thread, data element, and program point of the failure, offering dense, actionable feedback to the AI agent.
- Agentic Optimization Harness with ICRL: An AI agent, powered by In-Context Reinforcement Learning (ICRL), uses the feedback from the invariant validation to iteratively improve kernel implementations. This agent leverages a curated knowledge base of GPU optimization techniques and learns to select optimizations and synthesize effective data-flow invariants. The ICRL process uses invariant violations and runtime performance as reward signals, guiding the agent to learn better optimization strategies.
Datasets
The paper evaluates ARGUS on three key kernel families that dominate LLM inference:
- GEMM (General Matrix Multiplication): Standard square matrix multiplications.
- Flash Attention: A highly optimized attention mechanism.
- Mixture-of-Experts (MoE): Feed-forward networks used in LLMs.
Additionally, ARGUS is evaluated on 200 KernelBench tasks, which are described as Level 1 and Level 2 problems, suggesting a benchmark suite for evaluating kernel generation.
Key Insights and Ideas
- Data-Flow Invariants as Global Constraints: The paper highlights that performance-critical GPU optimizations require reasoning about global data choreography, not just local correctness. Data-flow invariants provide a powerful mechanism to express and enforce these global constraints.
- Dense Feedback for AI Agents: Traditional unit tests offer sparse, binary feedback (pass/fail). ARGUS's compile-time invariant violations provide dense, structured feedback (counterexamples) that are far more effective for guiding AI agents in complex optimization tasks.
- Bridging the Abstraction Gap: The ARGUS DSL exposes hardware features and compiler policies without overwhelming LLMs with low-level details, striking a balance between expressivity and learnability.
- Compile-Time Validation is Key: Ensuring optimizations are correct before runtime is crucial. ARGUS's zero-overhead compile-time analysis prevents costly runtime errors and guides the optimization process effectively.
Key Results
- Near Hand-Optimized Performance: ARGUS-generated kernels achieve 99-104% of the effective throughput of state-of-the-art hand-optimized assembly implementations on AMD MI300X GPUs for GEMM, flash attention, and MoE kernels.
- Massive Speedups over Existing Agents: ARGUS delivers geometric-mean speedups of 2-1543× compared to existing agentic systems across the evaluated workload families.
- Generalization: ARGUS successfully generates correct kernels for 100% of Level 1 and 90% of Level 2 KernelBench tasks.
- Ablation Studies: Bank conflict mitigation was the most significant single contributor to performance in flash attention, followed by instruction scheduling.
Limitations and Future Work
- Algorithmic Optimizations: Data-flow invariants do not address fundamental algorithmic changes (e.g., switching from standard attention to online softmax). These are still handled by the planner and knowledge base.
- Direct CUDA/Assembly Generation: The authors note that directly generating high-performance CUDA or assembly is challenging due to the complexity of ensuring correctness across many optimizations, which can exceed LLM context windows. ARGUS's DSL approach mitigates this.
- Hardware Porting: Porting ARGUS to new GPU architectures requires integrating with the target toolchain and extending the knowledge base with hardware-specific optimizations.
- Future Work: The paper suggests that a refined version could leverage polyhedral analysis for faster validation. They also note that while the DSL is designed for LLMs, a more token-efficient design could be explored. The potential for agents to autonomously propose new optimization techniques (beyond relying on a curated knowledge base) is also an area for future exploration.
Summary updated