Extrapolating Neural Simulators to Novel Physical Regimes

This direction is merged into another direction: out-of-distribution generalization (out-of-distribution extrapolation). Its last version stays readable and citable here.

Version 1 of 1110 papers121 conceptsfrozen 2026-10-04

Machine learning has fundamentally changed how we approach physical simulations, offering speedups of several orders of magnitude over traditional numerical solvers. However, these models often suffer from a form of "tunnel vision." A model trained to simulate fluid flow at a specific Reynolds number or a material under moderate stress often collapses when asked to predict behavior in extreme or unseen regimes.

The core challenge in the field is moving beyond simple interpolation—where a model fills in the gaps between training points—toward robust out-of-distribution (OOD) generalization, where a model understands the underlying physics well enough to extrapolate to entirely new environments.

The Interpolation Trap

Most neural surrogates are excellent at curve-fitting. If you show a model a thousand simulations of water flowing through a pipe at various speeds, it will accurately predict the flow for a speed it hasn't seen before, provided that speed lies within the training range.

However, as noted by Ali Rabeh and colleagues in Geometry Matters: Benchmarking Scientific ML Approaches for Flow Prediction around Complex Geometries, these models often fail to generalize when physical parameters like the Reynolds number fall significantly outside the training distribution. The model hasn't learned the laws of fluid dynamics; it has learned a specific statistical map of a narrow regime. When the physics changes—for instance, when flow transitions from laminar to turbulent—the statistical map becomes irrelevant, leading to non-physical predictions or massive increases in error.

Disentangling Constants from Dynamics

One reason models fail to extrapolate is that they conflate the state of a system (e.g., the current velocity of a fluid) with the physical constants governing it (e.g., its viscosity). If these are tangled together in a neural network's latent space, the model cannot "turn the knob" on a single physical parameter without breaking its understanding of the motion.

To solve this, Stathi Fotiadis and Mario Lino developed the SD-VAE (Supervised Disentangled Variational Autoencoder) in Disentangling ODE parameters from dynamics in VAEs. By using ground-truth physical parameters as a supervisory signal, the model explicitly separates the "what" (the dynamics) from the "how much" (the physical constants). This separation allows the model to adapt to novel configurations—such as physical setups it has never encountered—by simply adjusting the parameter branch of the network while keeping the dynamical core intact.

Preserving Structure Across Scales

Another barrier to generalization is scale. In physics, the same governing equations (like the Navier-Stokes equations) apply whether you are looking at a droplet or an ocean, but the dominant forces change. Standard neural networks often lack this "scale-consistency."

Researchers like Zongyi Li argue in Scale-Consistent Learning for Partial Differential Equations that models must be designed to respect the multi-scale nature of physics. Instead of treating a simulation as a flat image or a simple graph, architectures like the MultiScaleGNN developed by Mario Lino Valencia process information across multiple spatial resolutions. This allows the model to capture both local interactions and global constraints, preventing the "local vision" problem that causes standard models to fail when obstacles or domains grow in size.

From Physics-Informed to Physics-Preserved

The most common way to "teach" a model physics is through a physics-informed loss function (PINNs), which penalizes the model when it violates a known equation. However, this is a "soft" constraint; the model can still drift into non-physical territory during extrapolation.

A more robust approach is to "bake" the physics directly into the architecture. In Predicting parametric spatiotemporal dynamics by multi-resolution PDE structure-preserved deep learning, Hao Sun and Jian-Xun Wang 0001 introduce the PDE-Preserved Neural Network (PPNN). Rather than just hoping the model follows the rules, they embed discretized physical laws as fixed layers within the network. By combining these fixed physics layers with trainable high-resolution components, the model maintains long-term stability and generalizes to unseen physical parameters—like varying viscosity—far better than black-box models.

Benchmarking the Extrapolation Gap

As the field matures, the focus is shifting from "can we build a fast surrogate?" to "how far can this surrogate go?" This has led to the creation of standardized benchmarks like the Common Task Framework (CTF).

As described by Philippe Martin Wyder in Common Task Framework For a Critical Evaluation of Scientific Machine Learning Algorithms, these frameworks move beyond simple accuracy metrics. They specifically test a model's ability to extrapolate across different physical regimes, such as varying Reynolds numbers or flow intensities. By sequestering test sets that represent "extreme" physics, researchers can finally distinguish between models that have truly captured the underlying principles and those that are merely sophisticated interpolators.

Papers in this version

Showing 60 of 110 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_extrapolating_neural_simulators_to_novel_e4b6d135,
  title = {Extrapolating Neural Simulators to Novel Physical Regimes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extrapolating-neural-simulators-to-novel-physical-regimes/rdr_e4b6d13500c24a3eb8daeb14ad7c7858},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_extrapolating_neural_simulators_to_novel_e4b6d135_v1,
  title = {Extrapolating Neural Simulators to Novel Physical Regimes},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/extrapolating-neural-simulators-to-novel-physical-regimes/rdr_e4b6d13500c24a3eb8daeb14ad7c7858/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}