Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit

Junqi Yin, Mijanur Palash, M. Paul Laiu, Muralikrishnan Gopalakrishnan Meena, John Gounley, Stephen M. de Bruyn Kops, Feiyi Wang, Ramanan Sankaran, Pei ZhangarXiv 2025 · DOI · arXiv

Abstract

Simulating fluid turbulence at the viscous limit remains a foundational challenge in computational physics due to the extreme multiscale nature of energy dissipation. While deep learning models like Vision Transformers (ViTs) offer promise, they typically rely on patch-based tokenization that blurs critical small-scale eddies. We present a novel framework that achieves pixel-level resolution by combining a physics-inspired hierarchical architecture, the Hierarchical Turbulence Transformer (Matey), with a high-performance sequence parallelism algorithm, RingX. Our approach achieved a sustained 1.1 EFLOPS on the Frontier supercomputer with 94% scaling efficiency. By resolving features down to the Kolmogorov scale, this work marks a significant milestone in applying exascale AI to high-fidelity fluid dynamics.

Introduction: The Multiscale Challenge

Fluid turbulence is characterized by a cascade of energy from large-scale swirling flows down to trillions of tiny, energy-dissipating eddies. Capturing this "viscous limit" requires massive spatial resolution, often involving billions or trillions of grid points. In the context of deep learning, this presents a significant computational bottleneck. Standard Vision Transformers (ViTs) manage memory by grouping pixels into large patches, but this "patching" effectively acts as a low-pass filter, removing the high-frequency physical information necessary for accurate turbulence modeling.

Processing every pixel as an individual token would result in sequence lengths that exceed the memory and compute capacity of even the most advanced hardware, given the $O(S^2)$ complexity of standard attention mechanisms. To address this, we introduce a dual-pronged innovation: a hierarchical model structure that mirrors physical scales and an HPC-optimized attention algorithm.

The Hierarchical Turbulence Transformer (Matey)

The "Matey" architecture is designed to mimic the classical hierarchical approach to fluid dynamics, which categorizes flows into Reynolds-Averaged Navier-Stokes (RANS), Large Eddy Simulation (LES), and Direct Numerical Simulation (DNS) regimes.

The Hierarchical Turbulence Transformer (Matey) architecture, integrating multi-fidelity filters to resolve multiscale physics.

As shown in the figure above, the model splits the input into three fidelity modes:

  1. Low-fidelity: Captures global, long-range physics at a coarse resolution.
  2. Mid-fidelity: Focuses on intermediate scales within specific subdomains.
  3. High-fidelity: Resolves local, fine-scale physics at the individual pixel level.

By structuring the Transformer in this way, we reduce the effective sequence length from billions of tokens to a manageable few million, while ensuring that the smallest physical features—the dissipative eddies—are explicitly resolved rather than blurred.

RingX: Scaling Attention for HPC

While the hierarchical structure reduces the token count, exascale performance still requires highly efficient parallelization. Standard "Ring Attention" utilizes point-to-point communication to pass data between GPUs in a circular topology. However, this approach often fails to fully exploit the high-bandwidth collective communication primitives available on modern supercomputer interconnects.

We developed RingX, a sequence parallelism algorithm optimized for HPC environments like the Frontier supercomputer. RingX replaces simple point-to-point transfers with optimized collective communications, such as Broadcast and Allreduce, and improves the overlap between computation and communication.

Performance comparison demonstrating the speedup of RingX variants over standard Ring Attention across various GPU configurations.

Empirical results demonstrate that RingX provides up to a 2.5x speedup over standard Ring Attention. This efficiency was critical in achieving a sustained performance of 1.1 EFLOPS across 32,768 AMD GPUs, maintaining a 94% scaling efficiency.

Experimental Validation and Physical Accuracy

The framework was evaluated using two gold-standard turbulence benchmarks: Forced Homogeneous Isotropic Turbulence (HIT) and Stably Stratified Turbulence (SST).

In the HIT tests, which utilize $1024^3$ grid points from the Johns Hopkins Turbulence Database, the model's predictions for the energy spectrum and enstrophy (a measure of local rotation) almost perfectly matched ground-truth DNS data. Crucially, the model successfully captured the "dissipative range"—the scale at which turbulence dies out—a feat that previous AI models have struggled to achieve.

In SST cases, the model demonstrated an ability to generalize to complex interactions, such as buoyancy and the transition from laminar to chaotic flow. While there was a slight under-prediction of total energy in the initial time steps of stratified flows—likely due to the complexity of buoyancy flux—the overall physical fidelity remained remarkably high.

Conclusion and Future Directions

The success of the Matey architecture and RingX algorithm suggests a "Physics-AI Symmetry": the most effective AI models for scientific discovery are those that mirror the underlying structure of physical laws. By redesigning context parallelism for HPC hardware, we have shown that it is possible to resolve the smallest scales of turbulence at exascale.

Future work will focus on "in-situ" learning, where the AI model is trained concurrently with a physics simulation. This would eliminate the need for massive intermediate datasets and allow for even higher-resolution modeling. Furthermore, the hierarchical approach developed here holds potential for other multiscale scientific domains, including climate modeling and genomics.

Summary updated

Abstract

Turbulence plays a crucial role in multiphysics applications, including aerodynamics, fusion, and combustion. Accurately capturing turbulence's multiscale characteristics is essential for reliable predictions of multiphysics interactions, but remains a grand challenge even for exascale supercomputers and advanced deep learning models. The extreme-resolution data required to represent turbulence, ranging from billions to trillions of grid points, pose prohibitive computational costs for models based on architectures like vision transformers. To address this challenge, we introduce a multiscale hierarchical Turbulence Transformer that reduces sequence length from billions to a few millions and a novel RingX sequence parallelism approach that enables scalable long-context learning. We perform scaling and science runs on the Frontier supercomputer. Our approach demonstrates excellent performance up to 1.1 EFLOPS on 32,768 AMD GPUs, with a scaling efficiency of 94%. To our knowledge, this is the first AI model for turbulence that can capture small-scale eddies down to the dissipative range.
BibTeX
@misc{yin2025pixel,
  author = {Junqi Yin and Mijanur Palash and M. Paul Laiu and Muralikrishnan Gopalakrishnan Meena and John Gounley and Stephen M. de Bruyn Kops and Feiyi Wang and Ramanan Sankaran and Pei Zhang},
  title = {{Pixel-Resolved Long-Context Learning for Turbulence at Exascale: Resolving Small-scale Eddies Toward the Viscous Limit}},
  year = {2025},
  eprint = {2507.16697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.16697},
}
Figure