Mixture-of-Experts (MoE) models are the current darlings of the AI world, powering everything from Mixtral to DeepSeek. By only activating a fraction of their total parameters for any given token, they promise massive "brain power" without a massive compute bill. However, there is a catch: while the math is sparse, the system often isn't.
In a new paper from the Allen Institute for AI (Ai2) and the University of Washington, researchers reveal that standard training infrastructure—originally built for "dense" models like GPT-3—imposes a heavy "capacity tax" on MoE models. To fix this, they’ve overhauled the Olmo-core training stack, enabling efficient scaling up to a staggering 2.38 trillion parameters.
TL;DR
Researchers redesigned the Olmo-core training stack to eliminate the "capacity tax" that slows down Mixture-of-Experts models on traditional hardware. By moving tokens to experts (rather than weights to tokens) and using 8-bit precision (MXFP8), they achieved high efficiency at the 1.2 trillion parameter scale.
What Problem Does This Paper Address?
In a standard dense model, every GPU needs to see every weight. Frameworks like Fully Sharded Data Parallel (FSDP) handle this by "gathering" weights across the network whenever they are needed.
For MoE models, this is a disaster. An MoE model might have 100 experts, but a single token only uses two. Standard frameworks still move all 100 experts' weights around, creating a "capacity tax": your training speed drops as you add experts, even if the actual work per token stays the same. The authors also identify a "runtime tax"—the overhead of routing tokens, managing memory, and waiting for the CPU to tell the GPU what to do next.
Methods
The team moved away from the weight-sharding approach of FSDP toward a more surgical architecture:
- DDP + Expert Parallelism (EP): Instead of moving weights to the data, they keep expert weights resident on specific GPUs and move the tokens to the rank that owns the required expert.
- NVSHMEM Rowwise EP: They use a specialized communication library (NVSHMEM) to write tokens directly into an expert's buffer on a remote GPU. This avoids the "packing and unpacking" overhead typical of standard network calls.
- Synchronization-Free Training: They kept all routing metadata on the GPU. This prevents the "host wait" problem, where the powerful GPU sits idle waiting for the slower CPU to calculate how many tokens are going to which expert.
- Device-Scheduled Grouped GEMMs: Since experts receive different numbers of tokens (a "ragged" workload), they used specialized kernels (CUTLASS) that can handle uneven batches without wasting time on "padding" them to a uniform size.
- MXFP8 Integration: They utilized the new 8-bit microscaling format (native to NVIDIA Blackwell GPUs), which cuts data movement and compute costs in half compared to 16-bit formats.
Datasets
The researchers used the OLMo-mix-0625 dataset for their production runs and the C4 (Colossal Clean Crawled Corpus) for validation. They focused on "from-scratch" training rather than "upcycling" (converting a dense model to MoE), arguing that for long training horizons, starting from scratch leads to better expert specialization.
Key Insights and Ideas
- Token Gerrymandering: This is a fascinating "failure mode" the authors discovered. They found that MoE routers can actually "hack" the load-balancing loss. The loss value goes down (suggesting the model is balanced), but actual imbalance increases because the router learns to give "dead" experts a tiny bit of soft probability mass without actually sending them any tokens.
- Overlap is Not Free: A common trick in high-performance computing is to hide communication time by running it at the same time as computation. The authors found that this often causes "resource contention," where the two tasks fight for the GPU's internal bandwidth, potentially slowing down the kernels more than the time they were trying to save.
- One Shared Expert is Enough: Many modern MoEs use "shared experts" that every token visits. The authors mathematically prove that multiple ungated shared experts are functionally identical to one single, wider expert, simplifying the architecture.
Key Results
The redesigned Olmo-core stack demonstrated impressive scalability on NVIDIA B300 nodes:
- Trillion-Scale Efficiency: At 1.2 trillion parameters, the system reached 858 TFLOP/s/GPU using MXFP8.
- Scaling Range: The stack supported configurations from 12.9 billion to 1.2 trillion parameters across up to 512 GPUs.
- Capacity Test: In an experimental run using the DeepEP v2 backend, they successfully pushed the model size to 2.38 trillion parameters.
- Memory Savings: Activation recompute (recalculating values during the backward pass instead of storing them) removed nearly two-thirds of activation memory, allowing much larger models to fit on the same hardware.
Limitations and Future Work
- Random vs. Learned Routing: Most of the headline throughput numbers were measured using "random routing" as a baseline. In real training, a "learned" router can introduce imbalances that reduce these speeds by 1% to 9%.
- Hardware Specificity: Many of the biggest gains (like MXFP8) are specific to the latest NVIDIA Blackwell architecture.
- Post-Training: While the stack is optimized for pretraining, the authors note that post-training (like Reinforcement Learning from Human Feedback) introduces new challenges, such as the need for low-latency weight transfers to inference engines, which they have not yet fully benchmarked.
For the full technical deep-dive, you can find the paper on arXiv under the title "Supercharging Olmo-core for Efficient and Scalable MoE Training" (arXiv:2611.18956).
Summary updated