TL;DR
ShardTensor is an open-source PyTorch extension that introduces a novel domain parallelism paradigm, allowing massive scientific datasets to be sharded across multiple GPUs even at a batch size of less than one per device. By decoupling spatial data resolution from hardware memory limits, it enables high-fidelity training and inference for extreme-scale scientific machine learning (SciML) workloads like climate modeling and fluid dynamics.
What Problem Does This Paper Address?
Scientific Machine Learning (SciML) deals with high-dimensional, extreme-resolution data, such as 3D medical scans, petavoxel brain maps, and kilometer-scale climate simulations. Unlike large language models where parameter counts dominate memory consumption, SciML models suffer from an explosion of intermediate activations during training. Because the data volume scales exponentially with spatial dimensions ($2^N$), training at native resolution quickly hits a hard ceiling imposed by single-GPU memory limits—even with a batch size of one.
Existing distributed training frameworks like PyTorch's standard DTensor or Fully Sharded Data Parallel (FSDP) are designed for model weights and assume static, uniform tensor distributions. They cannot handle input and output data that dynamically change shapes across layers (e.g., via convolutions or pooling operations), leaving a critical gap for high-resolution SciML applications.
Methods
The authors introduce ShardTensor, an imperative, layer-by-layer abstraction extending PyTorch's DTensor. Crucially, ShardTensor adds "sharding shapes" metadata, allowing each device to track the exact local chunk size of a tensor along its sharded axis. This enables the framework to handle arbitrary, non-uniform chunking resulting from operations like convolutions or unstructured grids.
ShardTensor intercepts operations at runtime using PyTorch's dispatch mechanisms (__torch_function__, __torch_dispatch__, and @custom_op), dynamically injecting collective communications (such as "halo exchanges" to fetch edge pixels from neighboring GPUs) only when needed. This allows it to seamlessly compose with existing data and model parallelism on orthogonal GPU mesh axes.
Datasets
The framework was validated using a mix of synthetic benchmarks and real-world industrial applications:
- Synthetic Data: Used for baseline performance benchmarking of Ring Attention and Vision Transformers (ViT) in both 2D and 3D configurations.
- DrivAerML: An automotive aerodynamics dataset containing volumetric mesh fields, used to evaluate the Transolver architecture.
- GOES-16 & MRMS: A real-world dataset consisting of continental US (CONUS) satellite imagery and ground-based radar observations, used to train StormScope, a high-resolution weather forecasting model.
Key Insights and Ideas
- Activations, Not Parameters, are the Bottleneck: In SciML, the "curse of dimensionality" means doubling spatial resolution increases data size by $2^N$. Intermediate activations quickly dwarf model weights, making domain decomposition of the input data essential.
- Static vs. Dynamic Sharding: Standard distributed tensors assume static shapes because weights don't change size during a forward pass. Inputs do. ShardTensor's breakthrough is tracking per-rank sharding shapes dynamically, allowing it to support layers that alter spatial dimensions.
- Orthogonal Composability: Domain parallelism doesn't replace data or model parallelism; it complements them. By assigning domain sharding to an orthogonal axis of a 2D GPU mesh, researchers can scale batch size, model size, and data resolution simultaneously.
Key Results
- Vision Transformer (ViT) Scaling: For 2D ViT inference at $4096^2$ resolution, ShardTensor achieved a 15x speedup on 16 GPUs compared to a single GPU. For 3D training, it successfully scaled to over 1 billion input points using 16 GPUs.
- Memory Reduction: Peak memory usage for 2D ViT training followed a clean quadratic fit relative to spatial resolution, demonstrating that ShardTensor successfully distributes activation memory proportionally across devices.
- StormScope Weather Forecasting: Enabled continental-scale storm forecasting at a native 3 km resolution (tensor shape $1024 \times 1792$). The model required 114 GB of peak memory, which would fail on a single 80 GB H100 GPU, but trained stably when split across 2 GPUs using ShardTensor.
Limitations and Future Work
- Python Dispatch Overhead: The imperative, layer-by-layer dispatch model introduces Python-level latency. For small operations or low-resolution data, this overhead can outweigh parallelization benefits.
- Communication Bottlenecks: Operations like convolutions require halo exchanges, which can become a bottleneck on hardware with slower interconnects.
- Incomplete Compiler Integration: Integration with
torch.compileis not yet complete, meaning static-graph optimizations like kernel fusion and communication/computation overlapping are currently unavailable. - Operator Coverage: Not all PyTorch operations have native domain-parallel paths implemented today, occasionally forcing fallbacks to standard
DTensorsemantics.
Future work will focus on completing torch.compile integration and combining ShardTensor with activation checkpointing and CPU offloading for even deeper memory savings.
Summary updated