Machine learning-based modeling of physical systems has experienced a surge of interest in recent years. However, Scientific Machine Learning (SciML) has faced a "wild west" problem. Unlike Computer Vision, which has ImageNet to standardize evaluation, SciML has lacked a comprehensive, large-scale benchmark. Researchers often test their models on simple, low-dimensional problems that fail to reflect real-world complexity. Furthermore, generating high-quality training data requires significant domain expertise and compute time, creating a high barrier to entry.
PDEBench is a comprehensive, open-source benchmark suite designed to standardize and accelerate SciML. It provides 35 high-fidelity datasets across 11 different Partial Differential Equations (PDEs), along with standardized APIs and physics-informed metrics to evaluate how well ML models actually capture the underlying laws of nature.
The PDEBench Framework
To address the lack of standardization, the authors introduce a framework that includes:
- Standardized Data Access: A unified API to download and load datasets, making it as easy to use as standard ML libraries.
- Diverse Baselines: Pre-implemented versions of state-of-the-art models, including Fourier Neural Operators (FNO), U-Nets, and Physics-Informed Neural Networks (PINNs).
- Novel Metrics: Recognizing that standard error (RMSE) doesn't tell the whole story in physics, the authors introduced "Physics-view" metrics. These measure whether a model respects conservation laws (cRMSE), satisfies boundary conditions (bRMSE), and captures the correct frequency spectrum (fRMSE) in Fourier space.
Expansive Datasets
PDEBench is significantly more expansive than previous efforts, offering datasets derived from 11 well-known PDEs. It covers up to 10,000 simulation runs per PDE, providing the "big data" necessary for deep learning.
The benchmark spans a wide range of complexities:
- Classic 1D/2D Problems: Advection, Burgers’ equation (shock waves), and Diffusion-Reaction.
- Advanced Real-World Problems: Compressible and Incompressible Navier-Stokes (turbulence), Darcy Flow (porous media), and Shallow-Water equations (tsunamis/flooding).
- Dimensionality: Covers 1D, 2D, and 3D spatial domains.
Key Insights and Results
The introduction of PDEBench has already yielded several critical insights into the current state of SciML:
- RMSE is a Poor Proxy: A model might have a low Root Mean Square Error but still fail to capture critical small-scale features like turbulence or shock waves. The authors argue that we must evaluate models based on their physical consistency, not just their pixel-wise accuracy.
- The Resolution Challenge: ML models often struggle when moving from smooth, low-resolution data to high-resolution, "stiff" problems where the physics changes rapidly over small distances.
- Inverse Problems Matter: Beyond just predicting the future (forward problems), PDEBench evaluates "inverse problems"—the ability of a model to look at a final state and infer the initial conditions or hidden physical parameters (like viscosity) that caused it.
When evaluating baseline models, distinct performance trends emerged.
- FNO Dominance: In most forward-prediction tasks, the Fourier Neural Operator (FNO) outperformed U-Nets and PINNs, particularly in its ability to learn differential operators across different frequency regimes.
- Speed vs. Accuracy: Once trained, ML models were found to be significantly faster than classical solvers. For 3D Fluid Dynamics, inference took ~0.14 seconds compared to ~60 seconds for a traditional simulation—a massive speedup, though accuracy in turbulent regimes still lags behind.
- Generalization Gaps: The benchmarks revealed that current models still struggle with "temporal extrapolation"—predicting the behavior of a system far beyond the time window they were trained on.
Limitations and Future Work
While PDEBench is a major step forward, the authors acknowledge several limitations. Currently, the benchmark is limited to rectangular spatial domains, whereas real-world engineering often involves complex, irregular shapes (like air flowing over a car wing). Additionally, the current version does not yet cover multi-phase flows or quantum mechanics.
The team plans to extend the benchmark to include non-rectangular domains and more complex boundary conditions to further push the boundaries of what SciML can achieve.
Check out the code and data: https://github.com/pdebench/PDEBench
Summary updated