TL;DR
SS-HSDP is a novel training framework that decouples "runtime parameters" (optimized for high-performance compute kernels) from "update parameters" (optimized for flexible optimizer units). By using shared data views and hierarchical sharding, it achieves up to a 21.7% boost in training throughput (MFU) over existing state-of-the-art frameworks while maintaining or even reducing peak memory usage.
What Problem Does This Paper Address?
Training massive AI models requires "sharded data parallelism" (like ZeRO-3), where model weights, gradients, and optimizer states are split across multiple GPUs to save memory. However, there's a hidden conflict in how this data is organized:
- Compute Efficiency: High-performance kernels (like those for MXFP8 low-precision training) work best when data is "packed" into specific, contiguous layouts (e.g., combining Q, K, and V projections into one large matrix).
- Optimizer Flexibility: Modern optimizers like Muon operate on specific sub-units, such as individual attention heads, which might be scattered within those packed compute layouts.
Existing frameworks like PyTorch FSDP2 or Megatron-FSDP "couple" these two choices. If you want efficient compute, you lose optimizer flexibility; if you want flexible updates, you end up with redundant data copying and movement that kills performance. SS-HSDP aims to break this coupling.
Methods
The core innovation of SS-HSDP is Runtime–Update Decoupling. It treats the parameters used during the forward/backward pass (Runtime Parameters) as distinct from those used during the optimizer step (Update Parameters).
Key technical components include:
- Data Trees: A hierarchical memory management system where a single "root" allocation can be viewed through different "lenses." For example, the same memory can be seen as one large packed matrix for a GEMM kernel and as multiple independent heads for the Muon optimizer, without ever copying the data.
- Shard Chains: These manage how data is partitioned across GPUs at different levels (ZeRO-1, 2, and 3) while respecting the boundaries required by both compute kernels and optimizers.
- ZeroSM Communication: Standard communication libraries (like NCCL) often compete with compute kernels for GPU resources (Streaming Multiprocessors or SMs). SS-HSDP offloads data movement to dedicated hardware copy engines and uses host-driven Remote Memory Access (RMA) to minimize this contention.
- Fully Zero-Copy Execution: By aligning sharding boundaries with buffer layouts, the framework eliminates the need for intermediate "packing" or "copy-in/out" buffers during communication.
Datasets
The authors evaluated SS-HSDP on several large-scale, compute-bound workloads:
- Internal Video Models: Both Mixture-of-Experts (MoE) and Dense variants.
- Qwen3-30B-A3B: A large language model with an MoE architecture.
- MiMo (12L): A 73.5-billion parameter model variant. Evaluation was performed using both standard BF16 and advanced MXFP8 (microscaling) precision formats.
Key Insights and Ideas
- Decoupling is the "Silver Bullet": Separating the view of the data from its storage allows developers to use the best possible kernels and the most advanced optimizers simultaneously.
- Communication Contention is a Silent Killer: In compute-bound workloads, the mere presence of communication kernels on the GPU can slow down math kernels by over 50% due to resource contention. Offloading this (ZeroSM) is critical for scaling.
- Memory Reuse vs. Sharding: While retaining prepared weight shards for reuse increases persistent memory, "Outer ZeRO-1" sharding (partitioning optimizer states more aggressively) can more than offset this cost, leading to a net memory saving.
Key Results
- Throughput Gains: On 128 GPUs, SS-HSDP achieved a 21.7% higher Model FLOPs Utilization (MFU) than the fastest baseline (veScale-FSDP) on the MiMo model.
- Memory Efficiency: Despite its advanced features, it used 6.2% less peak memory per GPU than the most memory-efficient baseline on the same MiMo workload.
- Optimizer Support: It successfully integrated the Muon optimizer with only a 2.5–4.1% overhead, proving that sophisticated matrix-based optimizers can be used at scale without a massive performance penalty.
Limitations and Future Work
- Workload Sensitivity: The benefits of ZeroSM are most pronounced in large, compute-bound workloads. For very small workloads, the overhead of managing the offloaded communication might outweigh the gains.
- Implementation Complexity: The "Data Tree" and "Shard Chain" abstractions, while powerful, add a layer of complexity to the framework's internals compared to simpler sharding methods.
- Future Directions: The authors suggest further exploring the composition of SS-HSDP with other forms of parallelism, such as Tensor Parallelism or Expert Parallelism, to support even larger and more diverse model architectures.
Summary updated