SS-HSDP: Efficient and Flexible Sharded Data Parallelism via Runtime-Update Decoupling

Chendong Xiang, Pengle Zhang, Jinliang Li, Zichen Liang, Jintao Zhang, Fan Bao, Jun Zhu, Jianfei ChenarXiv 2026 · arXiv

TL;DR

SS-HSDP is a novel training framework that decouples "runtime parameters" (optimized for high-performance compute kernels) from "update parameters" (optimized for flexible optimizer units). By using shared data views and hierarchical sharding, it achieves up to a 21.7% boost in training throughput (MFU) over existing state-of-the-art frameworks while maintaining or even reducing peak memory usage.

What Problem Does This Paper Address?

Training massive AI models requires "sharded data parallelism" (like ZeRO-3), where model weights, gradients, and optimizer states are split across multiple GPUs to save memory. However, there's a hidden conflict in how this data is organized:

  1. Compute Efficiency: High-performance kernels (like those for MXFP8 low-precision training) work best when data is "packed" into specific, contiguous layouts (e.g., combining Q, K, and V projections into one large matrix).
  2. Optimizer Flexibility: Modern optimizers like Muon operate on specific sub-units, such as individual attention heads, which might be scattered within those packed compute layouts.

Existing frameworks like PyTorch FSDP2 or Megatron-FSDP "couple" these two choices. If you want efficient compute, you lose optimizer flexibility; if you want flexible updates, you end up with redundant data copying and movement that kills performance. SS-HSDP aims to break this coupling.

Methods

The core innovation of SS-HSDP is Runtime–Update Decoupling. It treats the parameters used during the forward/backward pass (Runtime Parameters) as distinct from those used during the optimizer step (Update Parameters).

Key technical components include:

  • Data Trees: A hierarchical memory management system where a single "root" allocation can be viewed through different "lenses." For example, the same memory can be seen as one large packed matrix for a GEMM kernel and as multiple independent heads for the Muon optimizer, without ever copying the data.
  • Shard Chains: These manage how data is partitioned across GPUs at different levels (ZeRO-1, 2, and 3) while respecting the boundaries required by both compute kernels and optimizers.
  • ZeroSM Communication: Standard communication libraries (like NCCL) often compete with compute kernels for GPU resources (Streaming Multiprocessors or SMs). SS-HSDP offloads data movement to dedicated hardware copy engines and uses host-driven Remote Memory Access (RMA) to minimize this contention.
  • Fully Zero-Copy Execution: By aligning sharding boundaries with buffer layouts, the framework eliminates the need for intermediate "packing" or "copy-in/out" buffers during communication.
Figure 4: Data structures and execution flow showing how Data Trees and Shard Chains enable decoupling.

Datasets

The authors evaluated SS-HSDP on several large-scale, compute-bound workloads:

  • Internal Video Models: Both Mixture-of-Experts (MoE) and Dense variants.
  • Qwen3-30B-A3B: A large language model with an MoE architecture.
  • MiMo (12L): A 73.5-billion parameter model variant. Evaluation was performed using both standard BF16 and advanced MXFP8 (microscaling) precision formats.

Key Insights and Ideas

  • Decoupling is the "Silver Bullet": Separating the view of the data from its storage allows developers to use the best possible kernels and the most advanced optimizers simultaneously.
  • Communication Contention is a Silent Killer: In compute-bound workloads, the mere presence of communication kernels on the GPU can slow down math kernels by over 50% due to resource contention. Offloading this (ZeroSM) is critical for scaling.
  • Memory Reuse vs. Sharding: While retaining prepared weight shards for reuse increases persistent memory, "Outer ZeRO-1" sharding (partitioning optimizer states more aggressively) can more than offset this cost, leading to a net memory saving.
Figure 5: Optimizations like batched updates and ZeroSM communication that reduce overhead and contention.

Key Results

  • Throughput Gains: On 128 GPUs, SS-HSDP achieved a 21.7% higher Model FLOPs Utilization (MFU) than the fastest baseline (veScale-FSDP) on the MiMo model.
  • Memory Efficiency: Despite its advanced features, it used 6.2% less peak memory per GPU than the most memory-efficient baseline on the same MiMo workload.
  • Optimizer Support: It successfully integrated the Muon optimizer with only a 2.5–4.1% overhead, proving that sophisticated matrix-based optimizers can be used at scale without a massive performance penalty.
Figure 6: Performance comparison showing SS-HSDP achieving higher MFU and lower memory usage across different models.

Limitations and Future Work

  • Workload Sensitivity: The benefits of ZeroSM are most pronounced in large, compute-bound workloads. For very small workloads, the overhead of managing the offloaded communication might outweigh the gains.
  • Implementation Complexity: The "Data Tree" and "Shard Chain" abstractions, while powerful, add a layer of complexity to the framework's internals compared to simpler sharding methods.
  • Future Directions: The authors suggest further exploring the composition of SS-HSDP with other forms of parallelism, such as Tensor Parallelism or Expert Parallelism, to support even larger and more diverse model architectures.

Summary updated

Abstract

Sharded data parallelism enables memory-efficient training of large models. While efficient training kernels favor packed, kernel-specific data layouts, optimizers may require variant data structures as optimization units. Existing sharded data-parallel frameworks couple these two choices, limiting runtime efficiency and optimization flexibility. To address this, we present \textbf{\our}, a training framework that separates runtime parameters from update parameters, allowing computation layouts and optimization units to be organized independently. Through shared data views and hierarchical sharding, \our supports this separation while avoiding redundant data movement. This design supports a broad range of optimizers, including matrix-based methods such as Muon, and performance optimizations such as fully zero-copy execution. Experiments on video and language model workloads with MXFP8 training show that \our achieves higher Model FLOPs Utilization (MFU) than FSDP2, veScale-FSDP, and Megatron-FSDP. At 128 GPUs, MFU improves by up to 21.7\% over the fastest evaluated baseline, with comparable or lower peak memory usage.
BibTeX
@misc{xiang2026ss,
  author = {Chendong Xiang and Pengle Zhang and Jinliang Li and Zichen Liang and Jintao Zhang and Fan Bao and Jun Zhu and Jianfei Chen},
  title = {{SS-HSDP: Efficient and Flexible Sharded Data Parallelism via Runtime-Update Decoupling}},
  year = {2026},
  eprint = {2611.16467},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16467},
}
Figure