A Fast, Programmable GPU-to-CPU Invocation Layer for GPU-Initiated Communication

Jiakun Yan, Shiksha Patel, David Sidler, Gengyu Rao, Brandon Potter, Bradford BeckmannarXiv 2026 · arXiv

TL;DR

KIWI is a high-performance framework that proves "reverse offload"—using CPU proxies to handle GPU networking—can actually outperform direct GPU-to-NIC communication (GDA). By providing a programmable callback interface and a ultra-low-latency queue, KIWI achieves sub-microsecond latencies and boosts end-to-end LLM serving performance by up to 29%.

What Problem Does This Paper Address?

Modern AI workloads, particularly Large Language Models (LLMs) using Mixture-of-Experts (MoE) architectures, require GPUs to communicate frequently and dynamically. Traditionally, developers have moved toward GPUDirect Async (GDA), where GPU threads drive the Network Interface Card (NIC) directly to avoid the "slow" host CPU.

However, the authors argue that GDA has hit a wall. Driving a NIC requires complex, sequential control logic (protocol selection, doorbell ringing, etc.) that runs poorly on the massively parallel, but sequentially weak, threads of a GPU. Furthermore, NICs have limited hardware endpoints; when thousands of GPU threads try to share them, contention and synchronization overhead skyrocket.

Fig. 1. GDA versus reverse offload. (a) With GDA, GPU threads drive the NIC themselves. (b) With reverse offload, the GPU hands each request to a CPU proxy, which drives the NIC. In both, the NIC moves payload data directly between GPU memories.

Methods

The authors introduce KIWI (Kernel-Initiated, Workload-Defined Invocations), a framework that brings the CPU back into the control path as a high-speed proxy.

Key Innovations:

  • Programmable Callbacks: Instead of a fixed set of networking primitives (like put or get), KIWI allows developers to register arbitrary C++ functions on the host as callbacks. A GPU kernel can then "invoke" these functions asynchronously.
  • Ultra-Fast Handoff: KIWI uses a specialized Single-Producer, Single-Consumer (SPSC) queue. A GPU "wave" (AMD's version of a warp) can publish a request to the CPU using a single 16-byte store over PCIe. This avoids expensive atomic operations and memory fences.
  • Decoupled Concurrency: Many GPU producers submit work to a small pool of dedicated CPU worker threads. This allows a few CPU cores to efficiently aggregate and drive NIC endpoints, solving the GDA scalability issue.
  • Smart Load Balancing: To prevent "hotspots" where one CPU proxy is overwhelmed while others are idle, KIWI uses a skewed round-robin mapping to distribute work from strided GPU requests evenly across proxies.
Fig. 2. Traditional reverse offload versus KIWI. (a) Traditional reverse offload hides the proxy and the NIC behind a fixed device API; all GPU threads share one queue, typically served by a single proxy thread. (b) KIWI exposes a single call, invoke, that runs user-defined callbacks on the CPU. Each wave owns its queue, several proxy threads poll the queues, and callbacks drive the NIC through any host library, such as LCI.

Datasets

The researchers evaluated KIWI on an AMD Instinct™ MI300X cluster with 400 Gb/s RoCE networking. They used:

  • KIWI-P2P: A prototype SHMEM-like library for point-to-point microbenchmarks.
  • KIWI-EP: Custom kernels for Expert-Parallel (EP) dispatch and combine operations.
  • vLLM Integration: Real-world serving of the GLM-5.3 model (a large MoE model) to measure Time Per Output Token (TPOT).

Key Insights and Ideas

  • CPUs are Better Controllers: The "aha" moment is realizing that for the branch-heavy, serial logic of networking protocols, a single fast CPU core is more efficient than hundreds of slow GPU threads.
  • Programmability = Performance: By moving the protocol logic to the CPU, the GPU kernel is simplified. For example, in MoE dispatch, a GPU can send one "multicast" request to the CPU, which then handles the fan-out to multiple nodes. This eliminates redundant control-path overhead on the GPU.
  • PCIe is the Floor: The latency of the GPU-to-CPU handoff in KIWI is so low (1.92 µs) that it is almost identical to a raw PCIe store. This proves that the software stack, not the hardware link, was the previous bottleneck for reverse offload.
The SPSC queue design that enables low-latency handoff

Key Results

  • Latency: KIWI-P2P achieves a small-message latency of 5.64 µs, which is 1.8x to 3.4x lower than state-of-the-art GDA implementations like MORI and rocSHMEM.
  • Throughput: At high concurrency (256 waves), KIWI's queue sustains 3.3x the throughput of the best existing alternatives.
  • MoE Performance: KIWI-EP kernels are up to 2.3x faster than existing expert-parallel backends on AMD GPUs.
  • End-to-End Serving: When integrated into vLLM for serving GLM-5.3, KIWI reduced the Time Per Output Token (TPOT) by up to 29% on an 8-node cluster.
Point-to-point latency comparison showing KIWI's advantage

Limitations and Future Work

  • CPU Overhead: KIWI requires dedicating a few CPU cores (e.g., 4 per GPU) to act as proxies. While host CPUs are often underutilized in GPU servers, this is still a resource trade-off.
  • OS Jitter: Because proxies run as standard OS threads, they are subject to preemption. The authors suggest that future work could involve dedicated OS scheduling support to make proxy execution more predictable.
  • Hierarchical Scaling: While KIWI-EP is highly efficient, the authors note that for massive clusters, they could further optimize by implementing hierarchical token routing (aggregating data at the node level before sending across the network).

The source code for KIWI is available at https://github.com/AMDResearch/kiwi.

Summary updated

Abstract

Modern GPU workloads increasingly rely on device-initiated communication to let GPU kernels communicate directly without synchronizing with the host at kernel boundaries. Existing libraries commonly favor placing transport control on GPU threads, an approach known as GPUDirect Async (GDA), and treat reverse offload to host proxy threads as a slower fallback. We argue that reverse offload can outperform GDA and should be a first-class communication path. We further propose exposing reverse offload as a programming interface that lets users define workload-specific communication operations. We present KIWI, a programmable reverse-offload framework that achieves sub-microsecond one-way device-to-host latency and hundreds of millions of operations per second. We address practical challenges of scaling reverse offload, including contention on shared queues, polling overhead, and load imbalance across proxies. KIWI-P2P, a SHMEM-like library built on KIWI, outperforms state-of-the-art GDA implementations in both latency and bandwidth. More importantly, KIWI-EP redesigns low-latency mixture-of-experts dispatch and combine kernels around workload-defined operations, lowering expert-parallel communication latency by up to 2.3x over the best existing backend on AMD GPUs and time per output token in vLLM by up to 29%. The source code of KIWI is available at https://github.com/AMDResearch/kiwi.
BibTeX
@misc{yan2026fast,
  author = {Jiakun Yan and Shiksha Patel and David Sidler and Gengyu Rao and Brandon Potter and Bradford Beckmann},
  title = {{A Fast, Programmable GPU-to-CPU Invocation Layer for GPU-Initiated Communication}},
  year = {2026},
  eprint = {2611.23848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23848},
}
Figure