ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants

Haohui Mai, Xiaoyan Guo, Xiangyun Ding, Daifeng Li, Qiuchu Yu, Chenzhun Guo, Cong Wang, Jiacheng Zhao, Christos Kozyrakis, Binhang YuanarXiv 2026 · DOI

TL;DR

This paper introduces ARGUS, a novel framework that uses "data-flow invariants" to guide AI agents in optimizing GPU kernels for LLM inference. By ensuring data is choreographed correctly throughout execution, ARGUS achieves performance close to hand-optimized libraries, significantly outperforming existing AI-based approaches.

What Problem Does This Paper Address?

Modern AI, especially Large Language Models (LLMs), relies heavily on powerful GPUs for computation. While AI agents can generate functionally correct GPU kernels, their performance often lags far behind manually optimized code. This performance gap is critical because even small improvements in kernel efficiency can translate to massive cost savings in large-scale deployments. The core challenge is that achieving peak GPU performance requires a deep, coordinated understanding of complex, low-level optimizations like tiling, memory staging, and instruction scheduling. Existing AI agents struggle with this because they typically rely on sparse feedback from unit tests, making it hard for them to diagnose and fix violations of global performance constraints.

Methods

ARGUS tackles this problem with a three-pronged approach:

  • Data-Flow Invariants via Tag Functions and Assertions: The core innovation is the concept of "data-flow invariants." These are compile-time specifications that describe how data should be managed throughout a kernel's execution. ARGUS uses a Pythonic Domain-Specific Language (DSL) with "tag functions" to attach symbolic annotations (like coordinates) to data elements. These tags propagate through computations. "Tag assertions" then enforce relational constraints on these tags at specific points in the code. For example, an assertion might ensure that two pieces of data expected to be paired for a matrix multiplication always carry matching tags.
  • Compile-Time Static Analysis: ARGUS employs a static analysis engine that verifies these data-flow invariants entirely at compile time, incurring zero runtime overhead. It uses abstract interpretation over a "layout algebra" and SMT solvers to check these invariants. Crucially, when an invariant is violated, the compiler provides concrete "counterexamples" that pinpoint the exact thread, data element, and program point of the failure, offering dense, actionable feedback to the AI agent.
  • Agentic Optimization Harness with ICRL: An AI agent, powered by In-Context Reinforcement Learning (ICRL), uses the feedback from the invariant validation to iteratively improve kernel implementations. This agent leverages a curated knowledge base of GPU optimization techniques and learns to select optimizations and synthesize effective data-flow invariants. The ICRL process uses invariant violations and runtime performance as reward signals, guiding the agent to learn better optimization strategies.

Datasets

The paper evaluates ARGUS on three key kernel families that dominate LLM inference:

  • GEMM (General Matrix Multiplication): Standard square matrix multiplications.
  • Flash Attention: A highly optimized attention mechanism.
  • Mixture-of-Experts (MoE): Feed-forward networks used in LLMs.

Additionally, ARGUS is evaluated on 200 KernelBench tasks, which are described as Level 1 and Level 2 problems, suggesting a benchmark suite for evaluating kernel generation.

Key Insights and Ideas

  • Data-Flow Invariants as Global Constraints: The paper highlights that performance-critical GPU optimizations require reasoning about global data choreography, not just local correctness. Data-flow invariants provide a powerful mechanism to express and enforce these global constraints.
  • Dense Feedback for AI Agents: Traditional unit tests offer sparse, binary feedback (pass/fail). ARGUS's compile-time invariant violations provide dense, structured feedback (counterexamples) that are far more effective for guiding AI agents in complex optimization tasks.
  • Bridging the Abstraction Gap: The ARGUS DSL exposes hardware features and compiler policies without overwhelming LLMs with low-level details, striking a balance between expressivity and learnability.
  • Compile-Time Validation is Key: Ensuring optimizations are correct before runtime is crucial. ARGUS's zero-overhead compile-time analysis prevents costly runtime errors and guides the optimization process effectively.

Key Results

  • Near Hand-Optimized Performance: ARGUS-generated kernels achieve 99-104% of the effective throughput of state-of-the-art hand-optimized assembly implementations on AMD MI300X GPUs for GEMM, flash attention, and MoE kernels.
  • Massive Speedups over Existing Agents: ARGUS delivers geometric-mean speedups of 2-1543× compared to existing agentic systems across the evaluated workload families.
  • Generalization: ARGUS successfully generates correct kernels for 100% of Level 1 and 90% of Level 2 KernelBench tasks.
  • Ablation Studies: Bank conflict mitigation was the most significant single contributor to performance in flash attention, followed by instruction scheduling.

Limitations and Future Work

  • Algorithmic Optimizations: Data-flow invariants do not address fundamental algorithmic changes (e.g., switching from standard attention to online softmax). These are still handled by the planner and knowledge base.
  • Direct CUDA/Assembly Generation: The authors note that directly generating high-performance CUDA or assembly is challenging due to the complexity of ensuring correctness across many optimizations, which can exceed LLM context windows. ARGUS's DSL approach mitigates this.
  • Hardware Porting: Porting ARGUS to new GPU architectures requires integrating with the target toolchain and extending the knowledge base with hardware-specific optimizations.
  • Future Work: The paper suggests that a refined version could leverage polyhedral analysis for faster validation. They also note that while the DSL is designed for LLMs, a more token-efficient design could be explored. The potential for agents to autonomously propose new optimization techniques (beyond relying on a curated knowledge base) is also an area for future exploration.

Summary updated

Abstract

LLM-based coding agents can generate functionally correct GPU kernels, yet their performance remains far below hand-optimized libraries on critical computations such as matrix multiplication, attention, and Mixture-of-Experts (MoE). Peak GPU performance requires coordinated reasoning over tightly coupled optimizations, including tiling, shared-memory staging, software pipelining, and instruction scheduling, while existing agents rely on sparse pass/fail feedback, leaving them unable to diagnose global constraint violations. We present Argus, an agentic framework that addresses this through data-flow invariants: compile-time specifications encoding how data must be choreographed throughout kernel execution. Argus introduces a tile-based, Pythonic DSL exposing hardware instructions and compiler policies while hiding low-level representations. The DSL provides tag functions to propagate symbolic annotations through data and control flow, and tag assertions to enforce relational constraints at use sites. When violations occur, the compiler returns concrete counterexamples identifying the thread, data element, and program point, enabling dense, structured feedback for targeted fixes. Invariants are verified at compile time via abstract interpretation over a layout algebra and SMT solving, with zero runtime overhead. An in-context reinforcement learning planner learns to select optimizations and synthesize effective invariants, supported by a curated knowledge base of GPU optimization techniques. We evaluate Argus on the AMD MI300X GPU across GEMM, flash attention, and MoE kernels accounting for over 90% of GPU time in LLM inference. Generated kernels achieve 99-104% of state-of-the-art hand-optimized assembly throughput and are 2-1543x faster than existing agentic systems. Argus further generalizes to 200 KernelBench tasks, solving 100% of Level 1 and 90% of Level 2 problems.
BibTeX
@misc{mai2026argus,
  author = {Haohui Mai and Xiaoyan Guo and Xiangyun Ding and Daifeng Li and Qiuchu Yu and Chenzhun Guo and Cong Wang and Jiacheng Zhao and Christos Kozyrakis and Binhang Yuan},
  title = {{ARGUS: Agentic GPU Optimization Guided by Data-Flow Invariants}},
  year = {2026},
  eprint = {2604.18616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18616},
}

Loading related content...

Figure