TL;DR
Weave is a novel runtime scheduling system for Mixture-of-Experts (MoE) large language model inference that dynamically optimizes GPU resource allocation. By adjusting the split of Streaming Multiprocessors (SMs) between computation and communication on a per-layer and per-GPU basis, Weave achieves a 2.89× MoE-layer speedup and eliminates the idle "pipeline bubbles" that plague existing static systems.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures scale up model capacity without a proportional increase in computational cost by routing tokens to only a sparse subset of specialized "expert" networks. In distributed settings, these experts are partitioned across multiple GPUs using Expert Parallelism (EP). This introduces a massive communication bottleneck: tokens must be sent to the correct GPUs (dispatch) and their outputs gathered back (combine).
As shown in Figure 1, modern GPU hardware scaling exacerbates this issue because raw tensor compute performance (TFLOPS) is growing much faster than interconnect (NVLink) bandwidth.
To hide this communication latency, state-of-the-art systems try to overlap communication and computation by splitting the GPU's SMs between the two tasks. However, existing approaches rely on fixed compile-time splits or coarse iteration-wide adjustments. Because MoE routing is inherently dynamic and highly skewed, these rigid policies lead to two types of waste:
- Spatial Mismatch: The optimal SM split varies across different layers and even different GPUs within the same layer. A fixed split means some GPUs waste compute throughput while others bottleneck on communication bandwidth.
- Temporal Bubbles: Strict data dependencies (e.g., expert computation cannot begin until dispatch finishes) create large idle gaps where communication SMs sit completely unused during the middle of execution.
Methods
The authors introduce Weave, the first MoE overlap system that performs fine-grained, dynamic SM scheduling at runtime. Weave operates inside a persistent megakernel that fuses all five stages of an MoE forward pass (dispatch, gemm0, activation, gemm1, and combine).
The core innovation relies on a key insight: although MoE workloads are dynamic, they become fully predictable immediately after the routing step of each layer. Weave exploits this exact moment by running a lightweight, on-GPU cost model in the megakernel prologue to jointly optimize execution along two dimensions:
- Spatial Scheduler: It calculates the exact volume of tokens to be sent and received per GPU. Using calibrated hardware throughput curves, it dynamically partitions the available SMs into communication workers and computation workers to perfectly match the workload ratio of that specific layer and GPU.
- Temporal Scheduler: To eliminate idle time, it introduces two mechanisms:
- Chunk Pipelining: It splits tokens into independent microbatches, spreading the combine communication phase into the middle of the GEMM computation phase for continuous overlap.
- Bubble Stealing: When communication workers finish their dispatch tasks and are waiting for chunked combine to start, they temporarily "steal" GEMM tiles to assist the computation workers, maximizing hardware utilization.
Datasets
The system was evaluated using workloads sampled from the ShareGPT dataset to simulate realistic routing inputs and token distributions. Additionally, the CodeSearchNet dataset was utilized during motivational experiments to analyze and quantify the performance gaps of existing baseline systems.
Key Insights and Ideas
- Exploiting Post-Routing Predictability: Instead of trying to predict routing distributions offline, Weave intercepts execution right after the gating mechanism determines token paths, turning a highly dynamic problem into a deterministic scheduling optimization problem at runtime.
- Consolidated Bubble Stealing: Rather than allowing SMs to constantly switch roles—which destroys cache locality and introduces control-flow overhead—Weave consolidates the stolen GEMM work into a single, clean window immediately following the dispatch phase.
- The Compute-Interconnect Divergence: Software-level scheduling is no longer just an optimization; it is a necessity. As hardware manufacturers struggle to scale communication bandwidth at the same rate as compute, intelligent scheduling inside the kernel is the primary way to unlock latent GPU performance.
Key Results
Weave was evaluated on a node with 4× NVIDIA H100 SXM GPUs across six mainstream MoE models (including DeepSeek-V3, Qwen3.5-35B, and Phi-3.5-MoE) at various sequence lengths.
- Layer-Level Speedup: Weave achieved a 2.89× geometric-mean speedup at the MoE-layer level compared to five state-of-the-art baselines (including DeepEP, Comet, and Triton-Distributed).
- End-to-End Speedup: It delivered a 1.33× geometric-mean end-to-end speedup across the evaluated models.
- Hardware Efficiency: Hardware profiling revealed that Weave achieved an exceptional 91.3% SM Active rate and a 47.1% compute-communication overlap ratio, successfully filling the pipeline bubbles that left prior systems underutilized.
Limitations and Future Work
The primary limitation acknowledged by the authors is hardware scale: Weave is currently implemented and validated on a single node containing 4× H100 GPUs with an Expert Parallelism degree of 4 (EP=4). The authors note that extending Weave's dynamic scheduling and cost models to larger GPU clusters and multi-node configurations—where communication bottlenecks are even more severe—is an important direction for future work.
Summary updated