In scientific computing, machine learning models have traditionally been treated as highly specialized experts. If you want to simulate fluid dynamics via the Navier-Stokes equations, you train a specific neural operator. If you switch to electromagnetics or structural mechanics, that model becomes entirely useless. It must be thrown away, and a new one must be trained from scratch.
This "specialist trap" exists because standard physics-informed neural networks (PINNs) and neural operators do not learn the underlying language of physics; instead, they memorize the specific mathematical mappings between a fixed set of boundary conditions, spatial grids, and differential operators. As noted by John Murray-Bruce et al. in Solving Inverse Source Problems for Sources with Arbitrary Shapes using Sensor Networks, this lack of physics-agnosticism severely limits the reuse of algorithms across diverse phenomena.
Recently, a powerful shift has emerged: the development of cross-domain foundation models for physical systems. Rather than overfitting to a single equation family, these models capture universal physical principles, enabling them to generalize across entirely different physical domains, functional forms, and geometries without retraining.
Abstracting Equations as Computational Graphs
Why do traditional neural solvers fail when a partial differential equation (PDE) changes? It is because the network's weights are implicitly conditioned on a fixed differential operator. If a coefficient changes or a new derivative term is added, the entire solution manifold shifts.
To break this limitation, researchers have begun treating the governing equations not as immutable laws hardcoded into the loss function, but as explicit inputs to the model.
Traditional Approach:
[Initial Conditions] ---> [Specialized Neural Solver] ---> [Predicted Evolution]
^
(Hardcoded Equation/Loss)
Foundation Approach:
[Initial Conditions] \
---> [Unified Graph/Transformer] ---> [Predicted Evolution]
[Equation as a Graph] /
In PDEformer: Towards a Foundation Model for One-Dimensional Partial Differential Equations, Zhanhong Ye et al. solve this by representing PDEs as computational graphs. In this framework, differential operators, coefficients, and boundary conditions are encoded as nodes and edges. A transformer architecture then processes this graph alongside the initial conditions. Because the transformer uses self-attention to dynamically route information between the field values and the operator nodes, it can solve a wide variety of PDE types in a zero-shot fashion. The model learns how a change in an equation's structure alters its physical propagation, achieving the same accuracy as specialized models without requiring a single gradient step of retraining.
Decoupling Geometry via Latent Dynamics
Another major barrier to cross-domain generalization is discretization. Fluid simulations might use a structured Eulerian grid, while astrophysics simulations rely on unstructured Lagrangian particles. Standard neural operators are tightly coupled to these geometric representations, failing when moved between irregular meshes or varying spatial resolutions.
To achieve true cross-domain capability, modern architectures decouple the geometric representation of the data from the underlying physical dynamics. They do this by projecting disparate physical inputs into a shared, continuous latent space.
[Eulerian Grid Data] ---> [Encoder A] \
---> [Shared Latent Space] ---> [Latent Dynamics Engine]
[Lagrangian Particle] ---> [Encoder B] /
This is the core intuition behind the framework introduced by Benedikt Alkin et al. in Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators. Their architecture compresses both grid-based and particle-based physical systems into a fixed-size latent space. The core simulation happens entirely within this latent domain, meaning the model's internal "dynamics engine" does not care whether the original source was smoke or plasma.
Similarly, Karn Tiwari et al. developed the Latent Mamba Operator (LaMO), which combines state-space models with kernel integral formulations in a compressed latent space to enable broad fine-tuning across diverse engineering applications. By operating in the latent space, these models abstract away the specific coordinate systems, allowing a single backbone to scale across fluid dynamics, weather forecasting, and complex engineering simulations.
On the geometric front, Haixu Wu et al. proposed Transolver: A Fast Transformer Solver for PDEs on General Geometries. Instead of calculating attention point-by-point across a physical mesh, Transolver adaptively groups mesh points into learnable, physics-aware "slices." This allows the network to extract intrinsic physical states that remain invariant whether the model is evaluating airflow over a car or a complex airfoil.
Meta-Learning the Shared Principles of Mechanics
Instead of memorizing specific trajectories, how can a model learn the deeper, invariant principles that govern all physical systems—such as the conservation of energy and momentum?
One approach is to leverage meta-learning to discover a "universal language" of physical interactions. In Towards Cross Domain Generalization of Hamiltonian Representation via Meta Learning, Yeongwoo Song et al. combine Graph Neural Networks with Model-Agnostic Meta-Learning (MAML). Instead of fitting the specific functional form of a single system, the model is trained to find a parameter initialization that can quickly adapt to any Hamiltonian system. Because it learns the broader geometric structure of classical mechanics (how positions and momenta exchange energy), it can generalize from a simple pendulum to chaotic magnetic mirrors with just a few data points.
This philosophy is scaled up significantly through multi-physics pre-training. As argued by Alberto Bietti et al. in Multiple Physics Pretraining for Spatiotemporal Surrogate Models, exposing neural surrogate models to a massive, diverse dataset of distinct physical phenomena forces the network to learn shared underlying features.
When a model is pre-trained on a vast zoo of equations, it develops a massive sample efficiency advantage. For example, Emmanuel de Bézenac et al. introduced Poseidon: Efficient Foundation Models for PDEs, a multiscale transformer foundation model. Because Poseidon captures shared physical representations across different types of partial differential equations, it can generalize to entirely new physics while requiring up to 50 times fewer samples than a traditional neural operator starting from scratch.
Low-Compute Adaptation and Cross-Modal Transfer
Building a foundation model from scratch requires immense computational resources. To bypass this, researchers are exploring two clever shortcuts: weight factorization and cross-modal transfer from large language models (LLMs).
Weight Factorization (SVD-PINNs)
Instead of fully retraining a network when the underlying physics or coefficients change, we can isolate and adapt only a fraction of the model's parameters. In SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition, Ka Chun Cheung et al. decompose the weight matrices of a pre-trained PINN using Singular Value Decomposition (SVD). They freeze the "basis" vectors—which capture the core spatial and temporal features of the physical domain—and only fine-tune the singular values. This allows the model to adapt to new coefficients or variations in the fundamental physics with minimal storage and highly stable training.
Repurposing Language Models (UPS)
Perhaps the most surprising mechanism for cross-domain physics generalization is cross-modal adaptation. Large Language Models are, at their core, highly optimized pattern-recognition engines designed to handle complex sequential dependencies.
In UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation, Junhong Shen et al. demonstrate that you can repurpose a pre-trained LLM to solve complex physical equations. By mapping continuous physical data into the latent token embedding space of a language model, the network's frozen attention layers can be used to propagate physical states over time.
[Physical Field Data] ---> [Linear Projection Layer] ---> [LLM Embedding Space] ---> [Frozen LLM Attention Layers] ---> [PDE Solution]
Because the LLM has already mastered abstract sequence modeling, this cross-modal alignment allows UPS to solve diverse 1D and 2D physics problems while requiring 4x less data and 26x less compute than models trained entirely from scratch.
The Path Forward
The transition from task-specific neural surrogates to universal physics foundation models mirrors the evolution seen in natural language processing and computer vision. By abstracting equations as graphs, operating in geometry-agnostic latent spaces, and leveraging multi-physics pre-training, the scientific machine learning community is moving closer to a single, unified model capable of simulating the physical world at scale.
Key Foundational Work
Solving Inverse Source Problems for Sources with Arbitrary Shapes using Sensor Networks (2017) — contributed 1 concept elements to this direction.
SVD-PINNs: Transfer Learning of Physics-Informed Neural Networks via Singular Value Decomposition (2022) — contributed 1 concept elements to this direction.