ShardTensor: Domain Parallelism for Scientific Machine Learning

Corey Adams, Peter Harrington, Akshay Subramaniam, Mohammad Shoaib Abbas, Jaideep Pathak, Mike Pritchard, Sanjay ChoudhryarXiv 2026 · arXiv

TL;DR

ShardTensor is an open-source PyTorch extension that introduces a novel domain parallelism paradigm, allowing massive scientific datasets to be sharded across multiple GPUs even at a batch size of less than one per device. By decoupling spatial data resolution from hardware memory limits, it enables high-fidelity training and inference for extreme-scale scientific machine learning (SciML) workloads like climate modeling and fluid dynamics.

What Problem Does This Paper Address?

Scientific Machine Learning (SciML) deals with high-dimensional, extreme-resolution data, such as 3D medical scans, petavoxel brain maps, and kilometer-scale climate simulations. Unlike large language models where parameter counts dominate memory consumption, SciML models suffer from an explosion of intermediate activations during training. Because the data volume scales exponentially with spatial dimensions ($2^N$), training at native resolution quickly hits a hard ceiling imposed by single-GPU memory limits—even with a batch size of one.

Existing distributed training frameworks like PyTorch's standard DTensor or Fully Sharded Data Parallel (FSDP) are designed for model weights and assume static, uniform tensor distributions. They cannot handle input and output data that dynamically change shapes across layers (e.g., via convolutions or pooling operations), leaving a critical gap for high-resolution SciML applications.

Methods

The authors introduce ShardTensor, an imperative, layer-by-layer abstraction extending PyTorch's DTensor. Crucially, ShardTensor adds "sharding shapes" metadata, allowing each device to track the exact local chunk size of a tensor along its sharded axis. This enables the framework to handle arbitrary, non-uniform chunking resulting from operations like convolutions or unstructured grids.

ShardTensor intercepts operations at runtime using PyTorch's dispatch mechanisms (__torch_function__, __torch_dispatch__, and @custom_op), dynamically injecting collective communications (such as "halo exchanges" to fetch edge pixels from neighboring GPUs) only when needed. This allows it to seamlessly compose with existing data and model parallelism on orthogonal GPU mesh axes.

Fig. 1. Dispatch architecture of ShardTensor.

Datasets

The framework was validated using a mix of synthetic benchmarks and real-world industrial applications:

  • Synthetic Data: Used for baseline performance benchmarking of Ring Attention and Vision Transformers (ViT) in both 2D and 3D configurations.
  • DrivAerML: An automotive aerodynamics dataset containing volumetric mesh fields, used to evaluate the Transolver architecture.
  • GOES-16 & MRMS: A real-world dataset consisting of continental US (CONUS) satellite imagery and ground-based radar observations, used to train StormScope, a high-resolution weather forecasting model.

Key Insights and Ideas

  • Activations, Not Parameters, are the Bottleneck: In SciML, the "curse of dimensionality" means doubling spatial resolution increases data size by $2^N$. Intermediate activations quickly dwarf model weights, making domain decomposition of the input data essential.
  • Static vs. Dynamic Sharding: Standard distributed tensors assume static shapes because weights don't change size during a forward pass. Inputs do. ShardTensor's breakthrough is tracking per-rank sharding shapes dynamically, allowing it to support layers that alter spatial dimensions.
  • Orthogonal Composability: Domain parallelism doesn't replace data or model parallelism; it complements them. By assigning domain sharding to an orthogonal axis of a 2D GPU mesh, researchers can scale batch size, model size, and data resolution simultaneously.

Key Results

  • Vision Transformer (ViT) Scaling: For 2D ViT inference at $4096^2$ resolution, ShardTensor achieved a 15x speedup on 16 GPUs compared to a single GPU. For 3D training, it successfully scaled to over 1 billion input points using 16 GPUs.
  • Memory Reduction: Peak memory usage for 2D ViT training followed a clean quadratic fit relative to spatial resolution, demonstrating that ShardTensor successfully distributes activation memory proportionally across devices.
GPU memory usage during ViT training as a function of spatial resolution
  • StormScope Weather Forecasting: Enabled continental-scale storm forecasting at a native 3 km resolution (tensor shape $1024 \times 1792$). The model required 114 GB of peak memory, which would fail on a single 80 GB H100 GPU, but trained stably when split across 2 GPUs using ShardTensor.

Limitations and Future Work

  • Python Dispatch Overhead: The imperative, layer-by-layer dispatch model introduces Python-level latency. For small operations or low-resolution data, this overhead can outweigh parallelization benefits.
  • Communication Bottlenecks: Operations like convolutions require halo exchanges, which can become a bottleneck on hardware with slower interconnects.
  • Incomplete Compiler Integration: Integration with torch.compile is not yet complete, meaning static-graph optimizations like kernel fusion and communication/computation overlapping are currently unavailable.
  • Operator Coverage: Not all PyTorch operations have native domain-parallel paths implemented today, occasionally forcing fallbacks to standard DTensor semantics.

Future work will focus on completing torch.compile integration and combining ShardTensor with activation checkpointing and CPU offloading for even deeper memory savings.

Summary updated

Abstract

Scientific Machine Learning (SciML) faces unique challenges for extreme-resolution data, with mitigations that often fail to scale or degrade the accuracy of trained models. While some specialized methods have achieved remarkable results in training models or performing inference on massive spatial datasets with bespoke techniques, there is no generalized framework for parallelization over input data below batch size one per device. In this work we introduce ShardTensor: a novel paradigm of domain parallelism that enables flexible scaling of input data to arbitrary sizes. By decoupling the spatial dimensionality of input data from hardware constraints, ShardTensor enables scientific machine learning workloads to reach new levels of high fidelity training and inference. We demonstrate both strong and weak scaling of workloads during training and inference, showing improved latency with strong scaling and demonstrating the capacity to process higher data sizes with weak scaling. Additionally, we demonstrate multiple dimensions of parallelization, removing barriers to SciML on extreme-scale inputs.
BibTeX
@misc{adams2026shardtensor,
  author = {Corey Adams and Peter Harrington and Akshay Subramaniam and Mohammad Shoaib Abbas and Jaideep Pathak and Mike Pritchard and Sanjay Choudhry},
  title = {{ShardTensor: Domain Parallelism for Scientific Machine Learning}},
  year = {2026},
  eprint = {2605.11111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.11111},
}

Loading related content...

Figure