TL;DR
GC-MoE is a high-performance inference operator that optimizes Mixture-of-Experts (MoE) models by physically partitioning GPU resources between communication and computation. By leveraging CUDA Green Contexts, it overlaps token dispatch with grouped GEMMs more efficiently than previous monolithic approaches, delivering up to a 1.35x speedup on NVIDIA Hopper and Blackwell GPUs.
What Problem Does This Paper Address?
In Large Language Model (LLM) serving, MoE architectures are the gold standard for scaling model capacity without ballooning compute costs. However, during the "decode" phase (generating text one token at a time), performance hits a wall. At small batch sizes, the MoE layer isn't limited by raw math power (Tensor Cores); instead, it's choked by dispatch latency (moving tokens between GPUs) and weight traffic (streaming massive expert weights from memory).
Existing state-of-the-art solutions, like DeepGEMM’s Mega MoE, use "monolithic fusion"—stuffing dispatch and computation into a single, giant kernel. While this allows some overlap, it forces communication and compute warps to fight over the same registers and shared memory within every Streaming Multiprocessor (SM). This resource contention and the need for expensive global synchronization barriers create significant overhead during interactive generation.
Methods
The authors introduce GC-MoE, which moves away from monolithic fusion toward a decomposed architecture using three key innovations:
- CUDA Green Contexts Partitioning: Instead of sharing every SM, GC-MoE uses the driver-level Green Contexts API to split the GPU. For example, on a Blackwell B300, it might reserve 20 SMs exclusively for a communication "dispatch" kernel and 128 SMs for a "compute" kernel. This provides hardware-level isolation and independent resource budgets for each task.
- Fine-Grained Token-Ring Handshake: A custom protocol allows the compute kernel to start working as soon as the first block of tokens arrives in a local "ring buffer," rather than waiting for the entire dispatch phase to finish.
- Persistent, Barrier-Free Synchronization: By using generation-tagged counters and double-buffered receive signals, GC-MoE eliminates the need for a "cross-rank barrier" (a slow round-trip communication between all GPUs) at the end of every layer.
- Dual-Architecture FP8/FP4 GEMM: The compute kernel is highly optimized for low-precision formats. It features a native hardware path for Blackwell and a specialized software-decoded path for Hopper, both supporting FP8 activations and FP4 weights.
Datasets
The system was evaluated using production-scale MoE configurations rather than standard training datasets:
- GLM-5.2: Evaluated on 8x NVIDIA B300 GPUs.
- DeepSeek-V4.1-Flash: Evaluated on both 8x NVIDIA H20 (Hopper) and 8x NVIDIA B300 (Blackwell) GPUs.
- The evaluation scanned a range of 1 to 256 tokens per rank, covering the most critical scenarios for interactive LLM serving.
Key Insights and Ideas
- The "Ceding SMs" Paradox: Because decode GEMMs are bandwidth-bound, the compute partition doesn't actually need all the SMs on the chip to reach peak performance. Giving up ~13% of the SMs to a dedicated dispatch partition costs almost zero compute throughput but drastically reduces communication latency.
- Spatial vs. Temporal Overlap: Traditional fused kernels attempt temporal overlap (running different tasks on the same hardware at different times). GC-MoE achieves spatial overlap at the device level, allowing each role to be tuned independently without perturbing the other's register or shared memory footprint.
- Graph-Aware Design: By organizing the layer as a three-node CUDA Graph, the system minimizes host-side overhead and ensures that the driver can schedule concurrent kernels deterministically.
Key Results
GC-MoE outperformed the state-of-the-art DeepGEMM baseline at all 54 measured data points:
- On 8x H20 (Hopper): Achieved a 1.188x average speedup for DeepSeek-V4.1-Flash, with peaks of 1.296x for routed experts and 1.353x when including fused shared experts.
- On 8x B300 (Blackwell): Achieved a 1.106x average speedup without shared experts and 1.123x with them (peaking at 1.209x).
- Numerical Accuracy: Maintained high precision, with a minimum cosine similarity of 0.9999997 against the baseline, ensuring that the performance gains don't come at the cost of model quality.
Limitations and Future Work
- Targeted at Decode: The benefits of SM partitioning are most pronounced in the bandwidth-bound decode phase. In the "prefill" phase (large batches), the compute partition becomes the bottleneck, and the speedup narrows significantly (1.01–1.13x).
- Static Constraints: The current implementation has hard design limits, such as a maximum of 72 GPU ranks and specific tile size requirements.
- Future Directions: The authors plan to explore adaptive partitioning (dynamically resizing the communication/compute split) and extending the approach to more model families and prefill-heavy workloads.
Summary updated