Compiling Specialized GPU-Initiated Networking

Javid Baydamirli, Ismayil Ismayilov, Kaan Oktay, Didem UnatarXiv 2026 · arXiv

TL;DR

Obaleth is a specialized compiler that integrates GPU-initiated networking directly into a kernel's intermediate representation (IR). By treating communication as a first-class citizen alongside computation, it eliminates the "library tax" of generic networking stacks, delivering up to 3.3× faster small message transfers and a 29% throughput boost for large-scale LLM serving.

What Problem Does This Paper Address?

In modern machine learning, we've gotten incredibly good at optimizing computation. Compilers like Triton and XLA generate highly specialized kernels for GEMMs and attention. However, communication has remained a "black box."

Currently, GPUs talk to each other through handwritten libraries like NCCL or NVSHMEM. Because these libraries are designed to be generic—supporting any thread geometry or transport—they are inherently conservative. Every time a kernel wants to send data, it pays a heavy "tax": it must atomically reserve queue slots, issue memory fences, and ring "doorbells" to notify the Network Interface Card (NIC).

As models move toward fine-grained parallelism (like Mixture-of-Experts or KV-cache streaming), these micro-overheads accumulate. A single 8-byte "put" operation might take only 0.7 µs at the hardware level, but library overhead can bloat that to 4.6 µs.

Methods

The authors introduce Obaleth, a compiler (implemented as an MLIR-based Triton plugin) that makes networking visible to the optimization pipeline.

RDMA as Intermediate Representation (IR)

Instead of calling an opaque library function, Obaleth represents communication using a layered IR. It starts with high-level semantic operations (put, signal, wait) and lowers them through a backend-agnostic "Queue IR" before generating hardware-specific code for NVIDIA (mlx5), AMD, or AWS EFA.

Co-Optimization

Because the compiler "sees" both the math and the networking, it can perform several novel optimizations:

  • Submission Coalescing: Instead of ringing the NIC doorbell for every single message, Obaleth groups independent transfers into a single hardware submission.
  • Publication Ordering: It reorders data payloads and signals so the NIC doesn't stall waiting for data to land before it can process the next signal.
  • Issuer Analysis: If the compiler can prove only one thread is writing to a queue, it removes expensive atomic operations.
  • Completion Planning: It analyzes buffer lifetimes to decide exactly when a "wait" for completion is actually necessary, rather than polling conservatively.
Figure 1. Posting a chain of commands. While a library pays a reservation, release, doorbell, and completion per call, Obaleth posts the chain with one of each.

Datasets and Benchmarks

The authors evaluated Obaleth across a variety of hardware and software configurations:

  • Hardware: NVIDIA H100 (4 GPUs/node), B200 (8 GPUs/node), and AMD MI355X.
  • Microbenchmarks: Point-to-point latency, message rate, and all-to-all bus bandwidth.
  • Real-world Workloads: Serving the Qwen3-235B and Qwen3-30B models within the vLLM framework, specifically targeting Mixture-of-Experts (MoE) dispatch and combine patterns.
  • Baselines: NCCL GIN, NVSHMEM, and DeepEP (a specialized MoE library).

Key Insights and Ideas

  • The "Library Tax" is avoidable: By moving networking into the compiler, you can prove that certain safety checks (like system-scope fences) are redundant for specific workloads.
  • NIC-side stalls are a hidden bottleneck: Even if the GPU issues commands quickly, the NIC can stall if signals are interleaved poorly with data. Obaleth's "Publication Ordering" pass specifically targets this NIC-side idle time.
  • Memory Efficiency: Generic libraries often pre-allocate massive "symmetric heaps" (hundreds of MiBs). Obaleth only allocates what the specific kernel needs, using ~12–26× less device memory.
  • Tiered Adoption: Users can use Obaleth as a drop-in library (Tier-1) or write custom Triton kernels (Tier-2) to fuse communication deeply with computation.
Figure 4. Obaleth’s optimization passes, described in §5.3, ap- plied cumulatively to two independent chains on one endpoint, removing per-call submissions, NIC holds, and atomics.

Key Results

Obaleth consistently outperforms industry-standard libraries:

  • Latency: On H100 GPUs, it sends small messages up to 3.3× faster than NVSHMEM.
  • Throughput: In large-scale all-to-all tests (256 GPUs), Obaleth achieved 1.25× higher bandwidth than NVSHMEM and 2.3× higher than NCCL GIN.
  • LLM Serving: When integrated into vLLM to serve Qwen3-235B, it answered requests up to 1.29× faster than DeepEP, which is itself a highly optimized, handwritten library.
  • Cross-Platform: On AMD MI355X GPUs, Obaleth was 1.24–2.5× faster than the vendor-provided rocSHMEM and RCCL libraries.
Figure 2. Obaleth versus NCCL and NVSHMEM: all-to-all on H100 GPUs.

Limitations and Future Work

  • Integration Effort: While Tier-1 is a drop-in, achieving the best performance (Tier-2) requires users to write or adapt Triton kernels, which may be a barrier for some developers.
  • Transport Scope: The current implementation focuses heavily on RDMA (InfiniBand/RoCE) and peer-memory (NVLink). While AWS EFA support is implemented, a full-scale evaluation on that platform is left for future work.
  • Static Shapes: Some optimizations rely on the communication pattern having a "static shape" that the compiler can analyze. Highly dynamic, data-dependent patterns might see fewer gains.

The code for Obaleth is available on GitHub.

Summary updated

Abstract

Machine learning systems specialize computation to the workload and hardware, but communication remains in generic libraries whose device code and transport decisions are not visible to high-level compilers. We present Obaleth, a compiler that makes GPU-initiated communication part of a kernel's intermediate representation, co-optimizes it alongside computation, and lowers it to specialized transport code. Obaleth plans completions around buffer lifetimes and optimizes memory ordering, synchronization, and batching across NVIDIA and AMD GPUs. On H100 GPUs, Obaleth sends small messages up to 3.3$\times$ faster, and on 256 GPUs runs all-to-all 1.25$\times$ and 2.3$\times$ faster than NVSHMEM and NCCL GIN, with much smaller memory and register use. In vLLM, it serves Qwen3-235B on 16 B200 GPUs and answers requests up to 1.29$\times$ faster than with DeepEP.
BibTeX
@misc{baydamirli2026compiling,
  author = {Javid Baydamirli and Ismayil Ismayilov and Kaan Oktay and Didem Unat},
  title = {{Compiling Specialized GPU-Initiated Networking}},
  year = {2026},
  eprint = {2611.18145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18145},
}
Figure