Fuse-All MoE Inference Kernel with Load Balancing on TPUs

Rupeng Liu, Bangsheng Tang, Zongwei Zhou, Chengji Yao, Yejin LeearXiv 2026 · arXiv

TL;DR

Fuse-All MoE is a specialized TPU kernel that eliminates the efficiency bottlenecks of Mixture-of-Experts (MoE) models by fusing the entire computation pipeline—from token collection to final output writes—into a single operation. By combining this fusion with "load-aware" scheduling, the authors significantly reduce the performance penalty caused by uneven expert workloads, achieving up to a 34.5% boost in serving throughput on Google’s TPUv7 hardware.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) architectures are the secret sauce behind many modern Large Language Models (LLMs). They allow models to have massive parameter counts while only activating a small fraction of "experts" for any given token, keeping the computational cost per token manageable.

However, implementing MoE efficiently on hardware like TPUs is notoriously difficult for two reasons:

  1. Data Movement: Standard implementations shuffle intermediate data back and forth between High Bandwidth Memory (HBM) and the chip’s local memory (VMEM). This "materialization" wastes precious time and bandwidth.
  2. Load Imbalance: The "router" that assigns tokens to experts doesn't care about hardware balance. If one expert is popular (a "hot" expert), the device hosting it becomes a bottleneck, while other devices sit idle waiting for it to finish.

The authors tackle these issues by co-designing the computation kernel and the communication strategy to ensure that hardware stays busy and data stays on-chip as much as possible.

Methods

The core of the paper is the Fuse-All MoE kernel, implemented using Pallas (a JAX-based kernel language).

1. The "Fuse-All" Pipeline

Instead of treating token collection, matrix multiplications (matmuls), and output writes as separate steps, the authors fuse them into a single "compute-and-return" pipeline.

  • On-Chip Residency: Intermediates from the gate, up-projection, and activation stages stay in VMEM.
  • Direct Writes: The kernel calculates the final destination address for each token and writes the result directly there via DMA (Direct Memory Access), bypassing the need for slow auxiliary "scatter" buffers.
This figure shows the fusion boundary where intermediate data stays on-chip (VMEM) and results are written directly to the final destination.

2. Load Balancing Mechanisms

To handle "hot" experts, the authors introduced several clever scheduling tricks:

  • Row Bucketing: If a tile is only partially full of tokens, the kernel uses a smaller "bucket" of computation to avoid wasting cycles on padding.
  • Intra-Chip Redistribution: Since a TPUv7 chip has two TensorCores (chipmates), the system can dynamically move an expert's workload from a slammed core to its idle neighbor.
  • Symmetric EP8TP2: For the "decode" phase (where batch sizes are small), they use a variant that splits every active expert across both cores of a chip, ensuring perfectly balanced execution within the chip.

Datasets

The paper focuses on kernel performance and system throughput rather than model training. Evaluation was performed using a synthetic MoE configuration modeled after state-of-the-art LLMs:

  • Model Specs: 256 total experts, top-8 routing (each token chooses 8 experts).
  • Hardware: A TPUv7 (Ironwood) 2x2x2 slice (8 physical chips, 16 TensorCore devices).
  • Baseline: Compared against Megablox, a standard high-performance Pallas-based MoE implementation.

Key Insights and Ideas

  • Decoupling Producer from Destination: The most critical insight is maintaining a "fixed token-slot destination." By pre-calculating exactly where a token’s result should go in memory, the system can move the computation of that token to any device without needing to change the output logic. This makes dynamic load balancing much simpler.
  • Regime-Specific Optimization: The authors realized that "prefill" (processing the initial prompt) and "decode" (generating tokens one by one) have different bottlenecks. Prefill is compute-heavy and prone to skewed routing, while decode is memory-bound. Their system switches strategies (e.g., using EP8TP2 for decode) to match the specific needs of each phase.
  • Low-Precision Efficiency: The kernel natively supports FP4 weights and FP8/BF16 activations, maximizing the throughput of the TPU’s matrix units (MXU).
Details of the packed weight representation and logical layout used to optimize memory access.

Key Results

  • Prefill Speedup: In a highly skewed workload where one device had significantly more work, the Fuse-All kernel reduced the runtime from 2.3x (relative to a balanced workload) down to just 1.09x.
  • Decode Latency: The symmetric EP8TP2 approach reduced kernel latency by up to 21.3% compared to a standard fused implementation.
  • End-to-End Throughput: When integrated into a full serving stack, the system improved total tokens-per-second by up to 34.5% over the Megablox baseline.

Limitations and Future Work

  • Upstream Exchange: The current design still handles the initial "input activation exchange" (moving tokens to the chips that own the experts) outside the fused kernel. Fusing this could further reduce overhead.
  • Coarse-Grained Cross-Chip Balancing: While intra-chip balancing is highly dynamic, cross-chip balancing currently moves "whole experts." The authors suggest "split-token" balancing (splitting a single expert's tokens across multiple chips) as a promising but more complex future direction.
  • Weight Transfer Costs: The current planner for redistribution doesn't explicitly calculate the time cost of moving weights between chipmates, which could be optimized in future iterations.

Summary updated

Abstract

Mixture-of-Experts (MoE) architectures activate only a subset of experts for each token, allowing large language models (LLMs) to scale parameter capacity without a proportional increase in computation per token. However, intermediate data movement and uneven expert loads limit the efficiency of MoE inference on TPUs. To address these bottlenecks, we introduce Fuse-All MoE, a TPU kernel design that jointly optimizes expert computation, output communication, and load balancing. The kernel fuses token collection, expert projections, gated activation, and output writes, retaining expert computation intermediates on chip. Fixed token-slot destinations allow scheduling and expert placement to change while preserving routing semantics. In prefill-decode (PD) disaggregated serving, Fuse-All MoE balances EP16 prefill workloads through scheduling and expert redistribution, while symmetric EP8TP2 shares active expert computation between the two TensorCore devices within each chip during decode. Our experiments show that Fuse-All MoE reduces kernel runtime for a measured skewed EP16 prefill workload from 2.3x to 1.09x the same fixed reference time for a balanced workload. Across 32-512 decode tokens, symmetric EP8TP2 reduces kernel latency by up to 21.3% relative to fused EP16. With accompanying serving stack optimizations, the integrated system improves serving throughput by up to 34.5% over the Megablox EP16 baseline.
BibTeX
@misc{liu2026fuse,
  author = {Rupeng Liu and Bangsheng Tang and Zongwei Zhou and Chengji Yao and Yejin Lee},
  title = {{Fuse-All MoE Inference Kernel with Load Balancing on TPUs}},
  year = {2026},
  eprint = {2611.17857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17857},
}
Figure