GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts

Yong Li, Kan Liu, Tao LanarXiv 2026 · arXiv

TL;DR

GC-MoE is a high-performance inference operator that optimizes Mixture-of-Experts (MoE) models by physically partitioning GPU resources between communication and computation. By leveraging CUDA Green Contexts, it overlaps token dispatch with grouped GEMMs more efficiently than previous monolithic approaches, delivering up to a 1.35x speedup on NVIDIA Hopper and Blackwell GPUs.

What Problem Does This Paper Address?

In Large Language Model (LLM) serving, MoE architectures are the gold standard for scaling model capacity without ballooning compute costs. However, during the "decode" phase (generating text one token at a time), performance hits a wall. At small batch sizes, the MoE layer isn't limited by raw math power (Tensor Cores); instead, it's choked by dispatch latency (moving tokens between GPUs) and weight traffic (streaming massive expert weights from memory).

Existing state-of-the-art solutions, like DeepGEMM’s Mega MoE, use "monolithic fusion"—stuffing dispatch and computation into a single, giant kernel. While this allows some overlap, it forces communication and compute warps to fight over the same registers and shared memory within every Streaming Multiprocessor (SM). This resource contention and the need for expensive global synchronization barriers create significant overhead during interactive generation.

Methods

The authors introduce GC-MoE, which moves away from monolithic fusion toward a decomposed architecture using three key innovations:

  1. CUDA Green Contexts Partitioning: Instead of sharing every SM, GC-MoE uses the driver-level Green Contexts API to split the GPU. For example, on a Blackwell B300, it might reserve 20 SMs exclusively for a communication "dispatch" kernel and 128 SMs for a "compute" kernel. This provides hardware-level isolation and independent resource budgets for each task.
  2. Fine-Grained Token-Ring Handshake: A custom protocol allows the compute kernel to start working as soon as the first block of tokens arrives in a local "ring buffer," rather than waiting for the entire dispatch phase to finish.
  3. Persistent, Barrier-Free Synchronization: By using generation-tagged counters and double-buffered receive signals, GC-MoE eliminates the need for a "cross-rank barrier" (a slow round-trip communication between all GPUs) at the end of every layer.
  4. Dual-Architecture FP8/FP4 GEMM: The compute kernel is highly optimized for low-precision formats. It features a native hardware path for Blackwell and a specialized software-decoded path for Hopper, both supporting FP8 activations and FP4 weights.
Figure 1: Execution timelines. (a) On B300 the baseline is DeepGEMM’s Mega MoE, one persistent fused kernel on all 148 SMs: after routing counts and a tag-1 cross-rank barrier, the dispatch warps stream tokens, scales, and routing metadata into a per-block token ring while the GEMM consumes each ring block as soon as it is full. The phases overlap, but on the same SMs, with the dispatch warps and their staging buffers carved out of every SM’s register file and shared memory; the tail is max(Tcombine, tclean + ttag2)—the workspace clean overlaps the combine, but the tag-2 barrier is still a cross-rank round trip on every replay. (b) GC-MoE runs role1 (dispatch) on the 20-SM communication context and role2 (grouped GEMM) plus role3 (combine) on the 128-SM compute context. Role2 starts when the first ring block is filled (dashed trigger), and parity-double-buffered receive counters remove the tag-2 barrier, so the next replay can start as soon as role3 completes (green band). Each role is an ordinary kernel with its own register, shared-memory, and synchronization budget, so the dispatch and GEMM paths can be optimized independently. Bars are illustrative; measured end-to-end speedups span 1.03–1.30× across the routed-expert sweeps and 1.01–1.35× including shared experts (Section 6.2). Partitions are the B300 values.

Datasets

The system was evaluated using production-scale MoE configurations rather than standard training datasets:

  • GLM-5.2: Evaluated on 8x NVIDIA B300 GPUs.
  • DeepSeek-V4.1-Flash: Evaluated on both 8x NVIDIA H20 (Hopper) and 8x NVIDIA B300 (Blackwell) GPUs.
  • The evaluation scanned a range of 1 to 256 tokens per rank, covering the most critical scenarios for interactive LLM serving.

Key Insights and Ideas

  • The "Ceding SMs" Paradox: Because decode GEMMs are bandwidth-bound, the compute partition doesn't actually need all the SMs on the chip to reach peak performance. Giving up ~13% of the SMs to a dedicated dispatch partition costs almost zero compute throughput but drastically reduces communication latency.
  • Spatial vs. Temporal Overlap: Traditional fused kernels attempt temporal overlap (running different tasks on the same hardware at different times). GC-MoE achieves spatial overlap at the device level, allowing each role to be tuned independently without perturbing the other's register or shared memory footprint.
  • Graph-Aware Design: By organizing the layer as a three-node CUDA Graph, the system minimizes host-side overhead and ensures that the driver can schedule concurrent kernels deterministically.
Figure 2: The architecture of a GC-MoE layer replay, showing how roles are mapped to disjoint Green Contexts and synchronized via a shared symmetric-memory workspace.

Key Results

GC-MoE outperformed the state-of-the-art DeepGEMM baseline at all 54 measured data points:

  • On 8x H20 (Hopper): Achieved a 1.188x average speedup for DeepSeek-V4.1-Flash, with peaks of 1.296x for routed experts and 1.353x when including fused shared experts.
  • On 8x B300 (Blackwell): Achieved a 1.106x average speedup without shared experts and 1.123x with them (peaking at 1.209x).
  • Numerical Accuracy: Maintained high precision, with a minimum cosine similarity of 0.9999997 against the baseline, ensuring that the performance gains don't come at the cost of model quality.

Limitations and Future Work

  • Targeted at Decode: The benefits of SM partitioning are most pronounced in the bandwidth-bound decode phase. In the "prefill" phase (large batches), the compute partition becomes the bottleneck, and the speedup narrows significantly (1.01–1.13x).
  • Static Constraints: The current implementation has hard design limits, such as a maximum of 72 GPU ranks and specific tile size requirements.
  • Future Directions: The authors plan to explore adaptive partitioning (dynamically resizing the communication/compute split) and extending the approach to more model families and prefill-heavy workloads.

Summary updated

Abstract

Under expert parallelism, mixture-of-experts (MoE) layers pay all-to-all token dispatch, grouped GEMMs, and a combine step in every layer. In decode each rank holds only 1-256 tokens: dispatch latency and weight traffic dominate while tensor-core utilization is inherently low. In fused operators such as DeepGEMM's Mega MoE, dispatch and computation overlap only by sharing every SM: dispatch warps compete with TMA/MMA and epilogue warps for registers, shared memory, and issue slots; every phase shares one CTA resource budget, synchronization spans the whole SM array, and every replay ends with a workspace-clean pass and a cross-rank barrier. We present GC-MoE, an open-source FP8/FP4 MoE inference operator for NVIDIA Hopper and Blackwell GPUs that uses CUDA Green Contexts to split the device into a small communication partition and a large compute partition. Each role gets its own kernel, resource budget, and synchronization domain, so dispatch and GEMM tune independently. A three-node CUDA Graph (four on Hopper with fused shared experts) launches the dispatch and grouped-GEMM kernels as concurrent roots; a token-ring handshake starts computation on the first filled ring blocks, and double-buffered receive counters remove the trailing cross-rank barrier between replays. The compute partition runs operand-swapped tcgen05 block-scaled MMA on Blackwell and register-A WGMMA with in-register FP4 decoding on Hopper, with fused SwiGLU, requantization, and optional shared experts. On 8 H20 GPUs GC-MoE accelerates DeepSeek-V4.1-Flash decode by 1.188x on average on both the routed and shared-expert suites (peaks 1.296x and 1.353x); on 8 B300 GPUs it accelerates GLM-5.2 and DeepSeek-V4.1-Flash by 1.106x (1.123x with fused shared experts, peak 1.209x), beating Mega MoE at all 54 measured points at matched accuracy. GC-MoE reserves only 20 of 148 SMs on B300 for dispatch.
BibTeX
@misc{li2026gc,
  author = {Yong Li and Kan Liu and Tao Lan},
  title = {{GC-MoE: Overlapping Expert-Parallel Token Dispatch with FP8/FP4 Grouped GEMM via CUDA Green Contexts}},
  year = {2026},
  eprint = {2611.04222},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04222},
}
Figure