TL;DR
Fuse-All MoE is a specialized TPU kernel that eliminates the efficiency bottlenecks of Mixture-of-Experts (MoE) models by fusing the entire computation pipeline—from token collection to final output writes—into a single operation. By combining this fusion with "load-aware" scheduling, the authors significantly reduce the performance penalty caused by uneven expert workloads, achieving up to a 34.5% boost in serving throughput on Google’s TPUv7 hardware.
What Problem Does This Paper Address?
Mixture-of-Experts (MoE) architectures are the secret sauce behind many modern Large Language Models (LLMs). They allow models to have massive parameter counts while only activating a small fraction of "experts" for any given token, keeping the computational cost per token manageable.
However, implementing MoE efficiently on hardware like TPUs is notoriously difficult for two reasons:
- Data Movement: Standard implementations shuffle intermediate data back and forth between High Bandwidth Memory (HBM) and the chip’s local memory (VMEM). This "materialization" wastes precious time and bandwidth.
- Load Imbalance: The "router" that assigns tokens to experts doesn't care about hardware balance. If one expert is popular (a "hot" expert), the device hosting it becomes a bottleneck, while other devices sit idle waiting for it to finish.
The authors tackle these issues by co-designing the computation kernel and the communication strategy to ensure that hardware stays busy and data stays on-chip as much as possible.
Methods
The core of the paper is the Fuse-All MoE kernel, implemented using Pallas (a JAX-based kernel language).
1. The "Fuse-All" Pipeline
Instead of treating token collection, matrix multiplications (matmuls), and output writes as separate steps, the authors fuse them into a single "compute-and-return" pipeline.
- On-Chip Residency: Intermediates from the gate, up-projection, and activation stages stay in VMEM.
- Direct Writes: The kernel calculates the final destination address for each token and writes the result directly there via DMA (Direct Memory Access), bypassing the need for slow auxiliary "scatter" buffers.
2. Load Balancing Mechanisms
To handle "hot" experts, the authors introduced several clever scheduling tricks:
- Row Bucketing: If a tile is only partially full of tokens, the kernel uses a smaller "bucket" of computation to avoid wasting cycles on padding.
- Intra-Chip Redistribution: Since a TPUv7 chip has two TensorCores (chipmates), the system can dynamically move an expert's workload from a slammed core to its idle neighbor.
- Symmetric EP8TP2: For the "decode" phase (where batch sizes are small), they use a variant that splits every active expert across both cores of a chip, ensuring perfectly balanced execution within the chip.
Datasets
The paper focuses on kernel performance and system throughput rather than model training. Evaluation was performed using a synthetic MoE configuration modeled after state-of-the-art LLMs:
- Model Specs: 256 total experts, top-8 routing (each token chooses 8 experts).
- Hardware: A TPUv7 (Ironwood) 2x2x2 slice (8 physical chips, 16 TensorCore devices).
- Baseline: Compared against Megablox, a standard high-performance Pallas-based MoE implementation.
Key Insights and Ideas
- Decoupling Producer from Destination: The most critical insight is maintaining a "fixed token-slot destination." By pre-calculating exactly where a token’s result should go in memory, the system can move the computation of that token to any device without needing to change the output logic. This makes dynamic load balancing much simpler.
- Regime-Specific Optimization: The authors realized that "prefill" (processing the initial prompt) and "decode" (generating tokens one by one) have different bottlenecks. Prefill is compute-heavy and prone to skewed routing, while decode is memory-bound. Their system switches strategies (e.g., using EP8TP2 for decode) to match the specific needs of each phase.
- Low-Precision Efficiency: The kernel natively supports FP4 weights and FP8/BF16 activations, maximizing the throughput of the TPU’s matrix units (MXU).
Key Results
- Prefill Speedup: In a highly skewed workload where one device had significantly more work, the Fuse-All kernel reduced the runtime from 2.3x (relative to a balanced workload) down to just 1.09x.
- Decode Latency: The symmetric EP8TP2 approach reduced kernel latency by up to 21.3% compared to a standard fused implementation.
- End-to-End Throughput: When integrated into a full serving stack, the system improved total tokens-per-second by up to 34.5% over the Megablox baseline.
Limitations and Future Work
- Upstream Exchange: The current design still handles the initial "input activation exchange" (moving tokens to the chips that own the experts) outside the fused kernel. Fusing this could further reduce overhead.
- Coarse-Grained Cross-Chip Balancing: While intra-chip balancing is highly dynamic, cross-chip balancing currently moves "whole experts." The authors suggest "split-token" balancing (splitting a single expert's tokens across multiple chips) as a promising but more complex future direction.
- Weight Transfer Costs: The current planner for redistribution doesn't explicitly calculate the time cost of moving weights between chipmates, which could be optimized in future iterations.
Summary updated