TL;DR
KIWI is a high-performance framework that proves "reverse offload"—using CPU proxies to handle GPU networking—can actually outperform direct GPU-to-NIC communication (GDA). By providing a programmable callback interface and a ultra-low-latency queue, KIWI achieves sub-microsecond latencies and boosts end-to-end LLM serving performance by up to 29%.
What Problem Does This Paper Address?
Modern AI workloads, particularly Large Language Models (LLMs) using Mixture-of-Experts (MoE) architectures, require GPUs to communicate frequently and dynamically. Traditionally, developers have moved toward GPUDirect Async (GDA), where GPU threads drive the Network Interface Card (NIC) directly to avoid the "slow" host CPU.
However, the authors argue that GDA has hit a wall. Driving a NIC requires complex, sequential control logic (protocol selection, doorbell ringing, etc.) that runs poorly on the massively parallel, but sequentially weak, threads of a GPU. Furthermore, NICs have limited hardware endpoints; when thousands of GPU threads try to share them, contention and synchronization overhead skyrocket.
Methods
The authors introduce KIWI (Kernel-Initiated, Workload-Defined Invocations), a framework that brings the CPU back into the control path as a high-speed proxy.
Key Innovations:
- Programmable Callbacks: Instead of a fixed set of networking primitives (like
putorget), KIWI allows developers to register arbitrary C++ functions on the host as callbacks. A GPU kernel can then "invoke" these functions asynchronously. - Ultra-Fast Handoff: KIWI uses a specialized Single-Producer, Single-Consumer (SPSC) queue. A GPU "wave" (AMD's version of a warp) can publish a request to the CPU using a single 16-byte store over PCIe. This avoids expensive atomic operations and memory fences.
- Decoupled Concurrency: Many GPU producers submit work to a small pool of dedicated CPU worker threads. This allows a few CPU cores to efficiently aggregate and drive NIC endpoints, solving the GDA scalability issue.
- Smart Load Balancing: To prevent "hotspots" where one CPU proxy is overwhelmed while others are idle, KIWI uses a skewed round-robin mapping to distribute work from strided GPU requests evenly across proxies.
Datasets
The researchers evaluated KIWI on an AMD Instinct™ MI300X cluster with 400 Gb/s RoCE networking. They used:
- KIWI-P2P: A prototype SHMEM-like library for point-to-point microbenchmarks.
- KIWI-EP: Custom kernels for Expert-Parallel (EP) dispatch and combine operations.
- vLLM Integration: Real-world serving of the GLM-5.3 model (a large MoE model) to measure Time Per Output Token (TPOT).
Key Insights and Ideas
- CPUs are Better Controllers: The "aha" moment is realizing that for the branch-heavy, serial logic of networking protocols, a single fast CPU core is more efficient than hundreds of slow GPU threads.
- Programmability = Performance: By moving the protocol logic to the CPU, the GPU kernel is simplified. For example, in MoE dispatch, a GPU can send one "multicast" request to the CPU, which then handles the fan-out to multiple nodes. This eliminates redundant control-path overhead on the GPU.
- PCIe is the Floor: The latency of the GPU-to-CPU handoff in KIWI is so low (1.92 µs) that it is almost identical to a raw PCIe store. This proves that the software stack, not the hardware link, was the previous bottleneck for reverse offload.
Key Results
- Latency: KIWI-P2P achieves a small-message latency of 5.64 µs, which is 1.8x to 3.4x lower than state-of-the-art GDA implementations like MORI and rocSHMEM.
- Throughput: At high concurrency (256 waves), KIWI's queue sustains 3.3x the throughput of the best existing alternatives.
- MoE Performance: KIWI-EP kernels are up to 2.3x faster than existing expert-parallel backends on AMD GPUs.
- End-to-End Serving: When integrated into vLLM for serving GLM-5.3, KIWI reduced the Time Per Output Token (TPOT) by up to 29% on an 8-node cluster.
Limitations and Future Work
- CPU Overhead: KIWI requires dedicating a few CPU cores (e.g., 4 per GPU) to act as proxies. While host CPUs are often underutilized in GPU servers, this is still a resource trade-off.
- OS Jitter: Because proxies run as standard OS threads, they are subject to preemption. The authors suggest that future work could involve dedicated OS scheduling support to make proxy execution more predictable.
- Hierarchical Scaling: While KIWI-EP is highly efficient, the authors note that for massive clusters, they could further optimize by implementing hierarchical token routing (aggregating data at the node level before sending across the network).
The source code for KIWI is available at https://github.com/AMDResearch/kiwi.
Summary updated