TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication

Weicheng Xue, Bingqiang Wang, Li Yuan, Huihui Zhou, Yonghong TianarXiv 2026 · arXiv

TL;DR

Hierarchical (two-hop) dispatch can optimize communication in Mixture-of-Experts (MoE) training, but its real-world benefits are heavily throttled by software implementation overhead. This paper introduces TerraceMoE, a communication cost model governed by strict, code-enforced "validation gates" that explicitly withdraw prediction capabilities when they fail, preventing misleading or overly optimistic end-to-end throughput claims.

What Problem Does This Paper Address?

Mixture-of-Experts (MoE) models scale parameter counts efficiently by activating only a few "experts" per token. However, because these experts are distributed across different accelerators, MoE layers require massive all-to-all communication operations that frequently dominate training step time.

To mitigate this, researchers often propose hierarchical dispatch: deduplicating tokens within a fast local domain (like a NVLink node or rack) before sending them across a slower inter-node fabric. Traditionally, engineers rely on simple hardware bandwidth ratios to decide if this optimization is worth the effort. The authors argue that this approach is fundamentally flawed. Software overheads—such as local tensor unpacking and extra collective calls—drastically shift the breakeven point, meaning a hierarchical strategy that looks great on a topology diagram can easily result in a net performance loss in production.

Methods

The authors introduce TerraceMoE, a cost model designed to rigorously screen hierarchical routing choices:

  • Saturating Hockney Form: The model prices individual all-to-all calls by combining direct measurements of fixed host/network latency ($\alpha$) with a saturating bandwidth term ($\beta_{eff}$) that accounts for degraded performance on smaller message sizes.
  • Accountable Validation Gates: Rather than reporting a single, easily smoothed "average accuracy" figure, the authors implement strict, code-enforced validation gates. If the model's predictions deviate beyond pre-specified thresholds, the model code physically locks and refuses to output a prediction.
  • T-Route Routing Constraint: To make two-hop hierarchical dispatch deterministic enough to model, the authors combine device-limited routing and grouped-expert routing. This bounds per-token fan-out and fixes per-group quotas at compile time.
The validation gate design and thresholds.

Datasets

The model was calibrated and validated using six specifically designed measurement corpora collected across two hardware platforms (Machine A, a unified-fabric supernode, and Machine B):

Summary of the six scored corpora used to validate the model.

Additionally, the authors evaluated the downstream impact of their routing constraints using a dedicated quality corpus (Q1), which involved training a 13.14B-parameter MoE model over 62.9 billion tokens.

Key Insights and Ideas

  • The "Software Tax" is Massive: While the theoretical breakeven bandwidth ratio for hierarchical dispatch is a mere 1.10 under zero-overhead conditions, incorporating a standard PyTorch arrival chain causes the effective breakeven ratio to skyrocket to 3.98.
  • Hardware Ratios Lie: A cluster's raw interconnect specifications are insufficient for predicting MoE performance. The actual utility threshold shifts dynamically based on how the arrival chain is implemented and the hidden width ($H$) of the model.
  • Overlap Models are a Dead End: The authors attempted to use global single-parameter models to account for how communication overlaps with compute at the step level. They discovered this is a structural dead end; overlap exposure is a property of the execution schedule, not a static model constant.

Key Results

  • Communication-Level Success, Step-Level Failure: The model successfully cleared its communication-level gates (Tier-1), demonstrating highly accurate isolated collective modeling. However, it failed its step-level gate (Tier-2) due to the complexities of compute-communication overlap. True to their philosophy, the authors let the failure stand and explicitly refused to make unverified end-to-end training throughput predictions.
  • Low Quality Cost for T-Route: The T-Route constraint proved highly viable for training. It incurred a tiny validation loss penalty of only +0.0034 nats, safely below the pre-specified acceptable margin of 0.1 nats.

Limitations and Future Work

  • Scale Limits: The model's predictive power is strictly bounded to architectures up to 128 ranks. Data collected past 256 and 512 ranks proved too noisy and low-confidence to support robust claims.
  • Uniformity Assumption: TerraceMoE assumes that all accelerators within a fast domain share identical link bandwidths. This means it cannot accurately model non-uniform topologies, such as the heterogeneous node graphs found in AMD-based systems like Frontier.
  • Future Directions: The authors emphasize the need for an overlap-aware step instrument to safely unlock end-to-end step time predictions, alongside expanding the model to support edge-level cost terms for non-uniform hardware.

Summary updated

Abstract

Hierarchical two-hop dispatch can reduce slow-fabric traffic in expert-parallel Mixture-of-Experts training, but it adds a second collective and an arrival-side operator chain. We present a cost model for screening that trade at the communication-call level, bounded by validation gates that withdraw a capability in code when they fail rather than reporting a caveat. At a reference geometry with 16 groups of 8 ranks, $q=3$, $H=2048$, and 4096 tokens per rank, the corrected effective breakeven hierarchy ratio is 3.98 for the measured PyTorch arrival chain, 1.49 for a hypothetical fused target, and 1.10 at zero implementation overhead. These are ratio-only sensitivity results, not deployment predictions: platform A measures 1.03, platform B has no separated fast/slow measurement, and neither machine measured here reaches the hierarchical regime. Four communication-level corpora pass their gates; a drift probe and the step-level gate fail. The latter failure is enforced in code, so we make no training-throughput prediction. The enabling routing constraint fixes per-token fan-out and per-selected-group quota, while aggregate per-peer counts remain data-dependent. Its measured validation-loss cost is small but nonzero (+0.0034 nats); downstream equivalence is reported with incomplete estimator provenance and is therefore not independently reconstructible from the artifact. Code, calibration constants and the validation gates are at https://github.com/weich97/TerraceMoE-simulator.
BibTeX
@misc{xue2026terracemoe,
  author = {Weicheng Xue and Bingqiang Wang and Li Yuan and Huihui Zhou and Yonghong Tian},
  title = {{TerraceMoE: A Cost Model for Hierarchical MoE All-to-All Communication}},
  year = {2026},
  eprint = {2608.27874},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.27874},
}
Figure