TL;DR
The authors present Tessera, a distributed runtime that enables equivariant message-passing Transformers to scale to tens of millions of atoms on hundreds of GPUs. By co-designing the algorithm and the system—specifically through space-filling curve partitioning and communication-compute overlap—they achieve ab initio levels of accuracy at a scale previously reserved for much simpler empirical models.
What Problem Does This Paper Address?
In molecular dynamics (MD), researchers face a classic "accuracy vs. scale" trade-off. First-principles methods like Density Functional Theory (DFT) are highly accurate but computationally expensive ($O(N^3)$), limiting them to a few thousand atoms. Conversely, empirical force fields can simulate millions of atoms but fail to capture complex chemistry like bond breaking or charge transfer.
Equivariant Graph Neural Networks (GNNs) offer a middle ground: they learn the accuracy of DFT at a linear cost. However, distributing these models across many GPUs is notoriously difficult. Because atoms move, the "neighbor graph" (who interacts with whom) changes every step, leading to unpredictable communication patterns and variable tensor shapes that break standard optimization techniques like CUDA Graphs. Furthermore, the "message-passing" nature of these models requires multiple rounds of data exchange per simulation step, which quickly becomes a massive communication bottleneck.
Methods
The authors introduce Tessera, a runtime built on four key pillars of algorithm-system co-design:
- Spatial Partitioning via Hilbert Curves: Instead of dividing space into equal volumes (which leads to unbalanced atom counts), Tessera maps 3D coordinates onto a 1D Hilbert space-filling curve. It then cuts this curve into equal-count shards. This ensures every GPU handles the exact same number of atoms, fixing tensor shapes and allowing for efficient CUDA Graph capture. Spatial decomposition using Hilbert curves to fix tensor shapes.
- Multi-Level Neighbor Search: To find interacting atoms across devices, they use a three-level filter: a coarse axis-aligned bounding box (AABB) filter between devices, balanced query chunks within a device, and an exact search using the FAISS library. The three-level neighbor search pipeline.
- Communication-Compute Overlap: They split the attention mechanism into "local" and "remote" branches. By launching the remote data exchange asynchronously, they can hide the communication latency behind the local computation. Overlapping remote communication with local computation.
- Decentralized MD Engine: Most MD codes rely on a "Rank 0" coordinator to manage the simulation loop, creating a serial bottleneck. Tessera decentralizes this, letting each GPU integrate the motion of its own atoms locally. Decentralized on-GPU dynamics to remove serial bottlenecks.
Datasets
The system was evaluated using several benchmarks:
- Bulk Water: Used for weak-scaling studies, growing from 10k to 60k atoms per GPU.
- Aqueous KCl Solution: A 10.24-million-atom salt solution used for long-term trajectory validation and energy drift measurements.
- Norovirus Capsid: A 10.06-million-atom heterogeneous system (protein-dense regions in water) to test the limits of the equal-count partitioning strategy.
Key Insights and Ideas
- "Equal-Count" is the Key to Replay: By forcing each GPU to own an identical number of atoms, the authors turned a dynamic problem into a static one. This allowed them to use CUDA Graphs to "replay" entire simulation intervals, bypassing the high overhead of the Python/CPU-side dispatcher.
- Locality Limits the Peer Set: Using the Hilbert curve ensures that the number of "neighboring" GPUs a single GPU must talk to grows sublinearly (roughly $K^{0.5}$) rather than linearly with the total number of GPUs ($K$). This is crucial for scaling to 512+ GPUs.
- Equivariance is Communication-Heavy: The paper highlights that equivariant features (vectors and tensors) are much larger than the simple scalars used in classical MD. Their "Sender-side projection" trick reduces this traffic by ~26% by projecting features into a more compact space before shipping them across the network.
Key Results
- Massive Scale: Sustained a simulation of 30.7 million atoms on 512 NVIDIA H100 GPUs.
-
High Efficiency: Achieved 89.7% weak-scaling efficiency. As the problem and machine size grew 16x, the time per step only increased by 1.11–1.29x. Weak scaling results showing high efficiency up to 512 GPUs.
-
Speedups: Reusing the neighbor graph and communication plan cut the full-step time by 1.87x.
- Physical Validity: In a 10 ps NVE simulation of a salt solution, the energy drift was a negligible $0.42 \mu eV/atom/ps$, and the structural properties perfectly matched a serial reference.
Limitations and Future Work
- Strong Scaling "Knee": The authors noted that strong scaling (keeping the problem size fixed while adding GPUs) hits a point of diminishing returns around 8k atoms per GPU, where allocator pressure and small batch sizes degrade performance.
- Heterogeneity Penalty: While the equal-count cut balances the number of atoms, it doesn't perfectly balance the number of edges (interactions). In the Norovirus capsid test, protein-dense regions had more edges per atom, leading to a 14.5% performance penalty compared to uniform water.
- Precision Trade-offs: Using TF32 precision provides a significant speedup but introduces some energy drift. Finding the perfect split between high-precision and low-precision operations remains an open challenge.
- Future Directions: The authors suggest that a distributed sort and a sparser request structure are the next logical steps to reach the billion-atom scale.
Summary updated