Abstract
This work adapts the Masked Autoencoder (MAE) framework, originally successful in computer vision, to the domain of Partial Differential Equations (PDEs). By training a Vision Transformer (ViT) to reconstruct physical fields from heavily masked spatiotemporal data (up to 90% missing), the authors develop a "physics-aware" encoder. This encoder learns rich, generalizable latent representations of fluid dynamics and heat transfer, significantly improving the performance and robustness of downstream neural solvers for tasks such as time-stepping and super-resolution.
Introduction
Neural solvers—AI models designed to solve complex physics equations—offer significant speed advantages over traditional numerical methods. However, they frequently suffer from "brittleness": a model trained for a specific fluid flow often fails when parameters like viscosity, grid resolution, or boundary conditions are slightly altered.
To move toward "foundation models" for physics, we require methods that learn the underlying structure of physical laws across diverse scenarios without requiring massive amounts of labeled data for every new problem. The challenge lies in finding a self-supervised objective that forces a neural network to "understand" physics rather than merely memorizing specific patterns. This paper proposes that masked reconstruction is a powerful objective for learning these physical invariants.
The MAE-PDE Architecture
The authors introduce MAE-PDE, which adapts the Masked Autoencoder architecture for spatiotemporal physical data through several key innovations:
- Masked Pretraining: 1D or 2D PDE solutions are partitioned into patches, with a staggering 75% to 90% of them masked. A ViT encoder processes only the visible patches, while a lightweight decoder attempts to reconstruct the entire field. This high masking ratio forces the model to learn global physical constraints and long-range dependencies to fill in the gaps.
- Lie Point Symmetry Augmentation: To improve generalization, the authors utilize physical symmetries (e.g., time/space shifts and Galilean boosts) to augment training data, ensuring learned representations respect fundamental physical invariances.
- Multi-Resolution Handling: A "Token" strategy is used to handle varying grid sizes. An embedder network projects coordinate grids into tokens, allowing the model to adapt to different discretizations without simple padding.
- Downstream Conditioning: Once pretrained, the encoder provides "context" to standard neural solvers like Fourier Neural Operators (FNOs) or Unets. The solver is conditioned on the MAE’s latent representation to improve its predictions.
Experimental Setup
The study utilizes a wide array of 1D and 2D PDE datasets to test the model's versatility:
- 1D PDEs: KdV-Burgers (primary pretraining), Heat, Advection, Wave, and Kuramoto-Sivashinsky (KS).
- 2D PDEs: Heat, Advection, and Burgers (pretraining), plus complex Navier-Stokes equations and 2D Smoke Buoyancy data for extrapolation testing.
- Benchmarks: MAE-PDE is compared against standard FNOs, Unets, and contrastive learning approaches.
Key Insights
- Physics as a Language: Much like BERT learns the logic of language by filling in missing words, MAE-PDE learns the logic of physics by filling in missing regions of a flow field.
- Latent Arithmetic: The latent space is semantically meaningful. The authors demonstrate "vector arithmetic" on physics: adding the latent vector of a Heat equation to that of an inviscid Burgers equation results in a representation that decodes into a viscous Burgers solution.
- Low Inductive Bias: Unlike "Physics-Informed" models that bake specific equations into the loss function, MAE has low inductive bias. This allows it to scale across heterogeneous datasets and learn a unified representation.
Results and Discussion
The pretrained MAE encoder significantly outperformed supervised CNNs and MLPs in regressing equation coefficients and classifying boundary conditions. Conditioning an FNO solver on MAE representations reduced error by 10-40% across various tasks compared to standalone FNOs. Furthermore, the model showed a surprising ability to reconstruct equations it never saw during pretraining, such as the Kuramoto-Sivashinsky equation, suggesting it learned general features of wave propagation.
In 2D Navier-Stokes simulations, MAE-conditioned models were far more robust to changes in viscosity and forcing terms than baseline solvers. The use of Lie Point Symmetry augmentation was shown to be a critical component in achieving these low error rates.
Conclusion and Future Directions
While MAE-PDE shows great promise, it still faces challenges with "out-of-distribution" boundary conditions and the computational cost of fine-tuning the full ViT encoder during solver rollouts. Additionally, extremely high masking ratios can sometimes lose fine-scale physical details like small turbulent eddies. Future work will focus on scaling this approach to even larger, more heterogeneous unlabeled physics datasets to move closer to a true foundation model for the physical sciences.
Summary updated