CUCo: An Agentic Framework for Compute and Communication Co-design

Bodun Hu, Yoga Sri Varshan, Saurabh Agarwal, Aditya AkellaarXiv 2026 · DOI · arXiv

Abstract

Custom CUDA kernel development is essential for maximizing GPU utilization in large-scale distributed LLM training and inference, yet manually writing kernels that jointly leverage both computation and communication remains a labor-intensive and error-prone process. Prior work on kernel optimization has focused almost exclusively on computation, leaving communication kernels largely untouched even though they constitute a significant share of total execution time. We introduce CUCo, a training-free agent-driven workflow that automatically generates high-performance CUDA kernels that jointly orchestrate computation and communication. By co-optimizing these traditionally disjoint components, CUCo unlocks new optimization opportunities unavailable to existing approaches, outperforming state-of-the-art baselines and reducing end-to-end latency by up to $1.57\times$.

Introduction

In large-scale AI systems, computation (GPU math) and communication (inter-GPU data transfer) are typically treated as disjoint operations. Standard "host-side orchestration" relies on the CPU to manage these steps sequentially, which introduces idle gaps and synchronization overhead. While modern device-side APIs allow GPUs to trigger their own communication, the design space for such kernels is immense and hardware-dependent. Developers must determine optimal data tiling, chunk sizes, and synchronization scopes to avoid deadlocks and maximize throughput—a task that is notoriously difficult to perform manually.

CUCo addresses this challenge by automating the co-design of compute and communication. It moves away from manual kernel engineering toward a hardware-aware, automated optimization framework that bridges the gap between these two critical components of distributed workloads.

The CUCo Framework

CUCo employs a training-free agentic workflow structured around three main pillars to navigate the complex design space of fused kernels.

The CUCo agentic workflow, featuring the Fast-Path Agent for correctness and the Slow-Path Agent for evolutionary performance optimization.

Structured Design Space

Rather than generating code blindly, CUCo defines a formal 5D design space: Backend, Placement, Sync Scope, Issuer, and Granularity. This grounding ensures the agent's reasoning remains within valid communication semantics.

Two-Stage Agentic Workflow

  1. Fast-Path Agent (Correctness): This agent performs a "Host-to-Device Transformation" on existing host-driven code. Its primary goal is to produce a functionally correct kernel that passes numerical checks, establishing a reliable baseline for further optimization.
  2. Slow-Path Agent (Optimization): This agent uses an evolutionary search loop where an LLM acts as a "mutation operator." It suggests code changes through diffs, rewrites, or crossovers. To ensure efficiency, it utilizes Multi-Island Populations to explore diverse strategies and a Cascade Evaluation system to prune non-compiling or failing kernels early.

A Meta-Summarizer guides the search by analyzing successful mutations from previous generations and providing high-level recommendations to the LLM.

Evaluation and Results

CUCo was evaluated on four representative multi-GPU workloads: FlashAttention with Context Parallelism, DeepSeek-V3 MoE Dispatch/Combine, KV-Cache Transfer, and GEMM + AllGather.

Latency comparison for GEMM + AllGather workloads, showing significant performance gains for CUCo (GIN + LSA) over host-side baselines.

Performance Gains

Across all workloads, CUCo achieved latency reductions ranging from 5.3% to 36.2% compared to standard NCCL baselines. In communication-heavy scenarios, the framework delivered up to a 1.57× speedup. These gains are largely attributed to "per-tile pipelining," which allows compute blocks to begin processing data as soon as it arrives rather than waiting for entire network transfers to complete.

Search Efficiency

The two-phase "Explore then Exploit" strategy proved highly effective, reaching near-optimal performance in just 3 generations. The Fast-Path agent provided a critical head start; without it, the evolutionary search spent approximately 25% of its budget simply resolving compilation errors.

Conclusion

CUCo demonstrates that an agentic, evolutionary approach can effectively automate the complex task of compute-communication co-design. By decomposing the problem into correctness and performance phases, the framework synthesizes kernels that outperform hand-optimized baselines. Future work will focus on expanding support for more diverse communication primitives and refining the "Evolve-Block" annotations to handle increasingly complex kernel structures.

Summary updated

Abstract

Custom CUDA kernel development is essential for maximizing GPU utilization in large-scale distributed LLM training and inference, yet manually writing kernels that jointly leverage both computation and communication remains a labor-intensive and error-prone process. Prior work on kernel optimization has focused almost exclusively on computation, leaving communication kernels largely untouched even though they constitute a significant share of total execution time. We introduce CUCo, a training-free agent-driven workflow that automatically generates high-performance CUDA kernels that jointly orchestrate computation and communication. By co-optimizing these traditionally disjoint components, CUCo unlocks new optimization opportunities unavailable to existing approaches, outperforming state-of-the-art baselines and reducing end-to-end latency by up to $1.57\times$.
BibTeX
@misc{hu2026cuco,
  author = {Bodun Hu and Yoga Sri Varshan and Saurabh Agarwal and Aditya Akella},
  title = {{CUCo: An Agentic Framework for Compute and Communication Co-design}},
  year = {2026},
  eprint = {2603.02376},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02376},
}
Figure