Supercharging Olmo-core for Efficient and Scalable MoE Training

Tianhua Tao, Akshita Bhagia, Dirk Groeneveld, Pete Walsh, Tyler Romero, Yashas Samaga, Jacob Morrison, Iz Beltagy, Taira Anderson, Noah A. Smith, Hannaneh HajishirziarXiv 2026 · arXiv

Mixture-of-Experts (MoE) models are the current darlings of the AI world, powering everything from Mixtral to DeepSeek. By only activating a fraction of their total parameters for any given token, they promise massive "brain power" without a massive compute bill. However, there is a catch: while the math is sparse, the system often isn't.

In a new paper from the Allen Institute for AI (Ai2) and the University of Washington, researchers reveal that standard training infrastructure—originally built for "dense" models like GPT-3—imposes a heavy "capacity tax" on MoE models. To fix this, they’ve overhauled the Olmo-core training stack, enabling efficient scaling up to a staggering 2.38 trillion parameters.

TL;DR

Researchers redesigned the Olmo-core training stack to eliminate the "capacity tax" that slows down Mixture-of-Experts models on traditional hardware. By moving tokens to experts (rather than weights to tokens) and using 8-bit precision (MXFP8), they achieved high efficiency at the 1.2 trillion parameter scale.

What Problem Does This Paper Address?

In a standard dense model, every GPU needs to see every weight. Frameworks like Fully Sharded Data Parallel (FSDP) handle this by "gathering" weights across the network whenever they are needed.

For MoE models, this is a disaster. An MoE model might have 100 experts, but a single token only uses two. Standard frameworks still move all 100 experts' weights around, creating a "capacity tax": your training speed drops as you add experts, even if the actual work per token stays the same. The authors also identify a "runtime tax"—the overhead of routing tokens, managing memory, and waiting for the CPU to tell the GPU what to do next.

Methods

The team moved away from the weight-sharding approach of FSDP toward a more surgical architecture:

  • DDP + Expert Parallelism (EP): Instead of moving weights to the data, they keep expert weights resident on specific GPUs and move the tokens to the rank that owns the required expert.
  • NVSHMEM Rowwise EP: They use a specialized communication library (NVSHMEM) to write tokens directly into an expert's buffer on a remote GPU. This avoids the "packing and unpacking" overhead typical of standard network calls.
  • Synchronization-Free Training: They kept all routing metadata on the GPU. This prevents the "host wait" problem, where the powerful GPU sits idle waiting for the slower CPU to calculate how many tokens are going to which expert.
  • Device-Scheduled Grouped GEMMs: Since experts receive different numbers of tokens (a "ragged" workload), they used specialized kernels (CUTLASS) that can handle uneven batches without wasting time on "padding" them to a uniform size.
  • MXFP8 Integration: They utilized the new 8-bit microscaling format (native to NVIDIA Blackwell GPUs), which cuts data movement and compute costs in half compared to 16-bit formats.

Datasets

The researchers used the OLMo-mix-0625 dataset for their production runs and the C4 (Colossal Clean Crawled Corpus) for validation. They focused on "from-scratch" training rather than "upcycling" (converting a dense model to MoE), arguing that for long training horizons, starting from scratch leads to better expert specialization.

Key Insights and Ideas

  • Token Gerrymandering: This is a fascinating "failure mode" the authors discovered. They found that MoE routers can actually "hack" the load-balancing loss. The loss value goes down (suggesting the model is balanced), but actual imbalance increases because the router learns to give "dead" experts a tiny bit of soft probability mass without actually sending them any tokens.
  • Overlap is Not Free: A common trick in high-performance computing is to hide communication time by running it at the same time as computation. The authors found that this often causes "resource contention," where the two tasks fight for the GPU's internal bandwidth, potentially slowing down the kernels more than the time they were trying to save.
  • One Shared Expert is Enough: Many modern MoEs use "shared experts" that every token visits. The authors mathematically prove that multiple ungated shared experts are functionally identical to one single, wider expert, simplifying the architecture.

Key Results

The redesigned Olmo-core stack demonstrated impressive scalability on NVIDIA B300 nodes:

  • Trillion-Scale Efficiency: At 1.2 trillion parameters, the system reached 858 TFLOP/s/GPU using MXFP8.
  • Scaling Range: The stack supported configurations from 12.9 billion to 1.2 trillion parameters across up to 512 GPUs.
  • Capacity Test: In an experimental run using the DeepEP v2 backend, they successfully pushed the model size to 2.38 trillion parameters.
  • Memory Savings: Activation recompute (recalculating values during the backward pass instead of storing them) removed nearly two-thirds of activation memory, allowing much larger models to fit on the same hardware.

Limitations and Future Work

  • Random vs. Learned Routing: Most of the headline throughput numbers were measured using "random routing" as a baseline. In real training, a "learned" router can introduce imbalances that reduce these speeds by 1% to 9%.
  • Hardware Specificity: Many of the biggest gains (like MXFP8) are specific to the latest NVIDIA Blackwell architecture.
  • Post-Training: While the stack is optimized for pretraining, the authors note that post-training (like Reinforcement Learning from Human Feedback) introduces new challenges, such as the need for low-latency weight transfers to inference engines, which they have not yet fully benchmarked.

For the full technical deep-dive, you can find the paper on arXiv under the title "Supercharging Olmo-core for Efficient and Scalable MoE Training" (arXiv:2611.18956).

Summary updated

Abstract

Mixture-of-Experts (MoE) models add parameters without proportionally increasing per-token compute. However, training cost also depends on bytes moved, memory held, and kernels launched, and many of those costs follow total parameter size rather than active compute. On infrastructure built for dense models, this mismatch is costly: in early baselines, throughput fell as stored expert parameter size grew, even though the useful work per token stayed nearly fixed. To close this gap, we present a redesigned training stack in Olmo-core, built for efficient, scalable MoE training. We replace the Fully Sharded Data Parallel (FSDP) of earlier Olmo-core versions with an architecture built on Distributed Data Parallel (DDP), and add Expert Parallelism (EP), Pipeline Parallelism (PP), a distributed optimizer, and activation recompute. We propose NVSHMEM-based rowwise EP, which writes each token directly into its expert's buffer, avoiding dispatch and combine permutation overhead. We make EP fully synchronization-free by keeping routing metadata on the GPU and using device-scheduled grouped GEMMs. Closely integrated MXFP8 cuts compute, activation-memory, and transport costs, and topology-agnostic checkpoints let the parallel layout change between runs. On NVL8 B300 nodes, the stack supports training configurations from 12.9 billion to 1.2 trillion total parameters on up to 512 GPUs, reaching 858 useful-model TFLOP/s/GPU at 1.2 trillion with MXFP8 and per-layer recompute; an experimental capacity test with the optional DeepEP v2 backend reaches 2.38 trillion. We also explain each design choice and share findings for other training systems; for example, we identify a failure pattern we call Token Gerrymandering, in which the MoE router can learn to hack the load-balancing loss, and find that overlapping communication with computation can slow the overlapped kernels by more than it hides.
BibTeX
@misc{tao2026supercharging,
  author = {Tianhua Tao and Akshita Bhagia and Dirk Groeneveld and Pete Walsh and Tyler Romero and Yashas Samaga and Jacob Morrison and Iz Beltagy and Taira Anderson and Noah A. Smith and Hannaneh Hajishirzi},
  title = {{Supercharging Olmo-core for Efficient and Scalable MoE Training}},
  year = {2026},
  eprint = {2611.18956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18956},
}
Figure