From Trajectories to Fields: The Dimensionality Leap
An Ordinary Differential Equation (ODE) describes how a system's state changes over a single independent variable—typically time. Visually, we can think of an ODE as tracking a single particle's trajectory through a state space. Machine learning frameworks like Neural ODEs, sparse regression, and deep state-space models have become exceptionally good at learning these trajectories from data.
However, real-world physical phenomena—such as fluid dynamics, heat propagation, and electromagnetic fields—do not exist at a single isolated point. They vary across both time and space. To model them, we must transition to Partial Differential Equations (PDEs). Instead of tracking a single trajectory, we are now tracking an entire continuous field as it deforms, flows, and evolves.
As noted by Yesom Park et al. in Learning PDE Solution Operator for Continuous Modeling of Time-Series, ODEs inherently lack the expressive capacity to capture these complex spatio-temporal correlations. Yet, scaling our existing machine learning toolkits from time-only ODEs to multi-dimensional PDEs introduces severe mathematical and computational bottlenecks. Understanding why these methods struggle to scale—and how researchers are overcoming these limitations—is the core focus of current physics-informed machine learning.
The Combinatorial Explosion of Spatial Derivatives
Why is discovering a PDE so much harder than discovering an ODE? The answer lies in the search space of differential operators.
In an ODE system, automated governing equation discovery methods search for a function $f(u)$ that matches the time derivative $\dot{u}$. The candidate library consists of polynomial terms or simple combinations of the state variables. When we move to PDEs, the library must expand to include spatial derivatives. In a simple one-dimensional (1D) spatial domain, we add terms like $\partial_x u$ and $\partial_{xx} u$. But real physical systems operate in three-dimensional (3D) space. Suddenly, the library must include $\partial_y u, \partial_z u$, cross-derivatives like $\partial_{xy} u$, and complex vector calculus operators like gradients, divergences, and curls.
ODE Search Space: [ u, u², sin(u), ... ] --> Scales with variables
1D PDE Search Space: [ u, u_x, u_xx, ... ] --> Scales with spatial derivatives
3D PDE Search Space: [ u, u_x, u_y, u_z, u_xx, u_xy, u_xz, ... ] --> Combinatorial Explosion
S Chandra Mouli et al. highlight in MetaPhysiCa: Improving OOD Robustness in Physics-informed Machine Learning that symbolic meta-learning frameworks struggle to scale precisely because of this massive expansion of the basis function search space. Similarly, compositional physics-informed models that excel at temporal sequences lack the inherent mechanisms to handle these multi-dimensional spatial dependencies, as argued by Thomas Baldwin-McDonald et al. in Compositional Modeling of Nonlinear Dynamical Systems with ODE-based Random Features.
Because of this complexity, much of the early success in data-driven PDE discovery has been confined to idealized 1D benchmarks. For instance, Saeideh Khatiry Goharoodi et al. demonstrated the successful discovery of the heat equation from physical induction motor sensors in Data-driven discovery of the heat equation in an induction machine via sparse regression, but noted that moving beyond 1D to capture radial or circumferential flows introduces steep scaling hurdles.
This dimensionality wall is a recurring theme across the literature:
- Junsheng Zeng et al. in Deep learning discovery of macroscopic governing equations for viscous gravity currents from microscopic simulation data point out that automated discovery methods are frequently restricted to 1D macroscopic descriptions, failing to capture full multi-dimensional spatial dependencies.
- Yuchen He 0001 et al. note in Group projected Subspace Pursuit for Identification of variable coefficient differential equations (GP-IDENT) that identifying equations with varying coefficients becomes dramatically more difficult when scaling from 1D domains to complex 3D geometries.
- Yogesh Chandrakant Mathpati et al. demonstrate stochastic PDE discovery in Discovering stochastic partial differential equations from limited data using variational Bayes inference, but emphasize that moving from 1D benchmarks to 2D or 3D systems remains an unproven computational challenge for real-world phenomena like climate patterns.
Breaking the Physics: Why Space Changes Conservation Laws
In ODE modeling, we can enforce physical priors like energy conservation using Hamiltonian Neural Networks. These networks exploit the symplectic geometry of time-evolution, ensuring that the total energy of the system remains constant as it moves along its trajectory.
But in a PDE, conservation laws are not just temporal; they are localized in space. Energy, mass, or momentum does not just change over time; it flows across spatial boundaries. To preserve these properties discretely, we cannot simply apply an ODE solver to a spatially discretized grid. Doing so ignores the underlying geometric structure of the field.
Symplectic (ODEs): d/dt [ Energy ] = 0 --> Global conservation over time
Multisymplectic (PDEs): d/dt [ Energy Density ] + d/dx [ Flux ] = 0 --> Localized space-time conservation
Brynjulf Owren et al. point out in Preserving multiple first integrals by discrete gradients that traditional discrete gradient methods fail to naturally handle the infinite-dimensional nature of PDEs. To truly preserve structure in spatio-temporal fields, we must transition from standard symplectic integrators to multisymplectic field theories, which treat space and time on equal geometric footings. As Brian K. Tran et al. explain in A Type II Hamiltonian Variational Principle and Adjoint Systems for Lie Groups, extending structure-preserving principles to PDEs requires multisymplectic integrators capable of managing simultaneous spatial and temporal variations.
Similarly, baking in physical symmetries (like rotational, translational, or scaling invariance) becomes vastly more complex when variables depend on both space and time. Jianke Yang et al. show in Symmetry-Informed Governing Equation Discovery that while symmetry constraints drastically simplify ODE discovery, extending them to PDEs is a necessary but mathematically daunting frontier required to model complex systems like fluid dynamics or electromagnetism. Without these spatial constraints, data-driven models easily learn physically impossible behaviors.
The Challenge of High-Dimensional Inference
When we train machine learning models on real-world data, we rarely observe the full state of a system perfectly. We often deal with noisy, partial observations.
In an ODE setting, we can use Gaussian Processes or nonlinear observers to map partial measurements to a latent state trajectory. For example, frameworks like those proposed by Ben Calderhead et al. in Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes or Mona Buisson-Fenet et al. in Recognition Models to Learn Dynamics from Partial Observations with Neural ODEs successfully reconstruct hidden states over time.
But in a PDE, managing derivatives across a massive multi-dimensional coordinate grid causes traditional joint inference methods to break down computationally. If a 1D grid requires $N$ points, a 3D grid requires $N^3$ points. Performing Bayesian inference or state estimation over this grid becomes intractable.
Jonathan Schmidt et al. note in A Probabilistic State Space Model for Joint Inference from Differential Equations and Data that state-space models for joint inference struggle to scale when moving from ODEs to high-dimensional spatial domains. This limitation extends to other downstream tasks:
- Output Estimation: Calculating certified safety bounds on system outputs via barrier functionals is typically restricted to 1D spatial domains; extending them to 3D engineering problems requires adapting complex integral inequalities to handle the increased geometric complexity, as shown by Mohamadreza Ahmadi et al. in Barrier Functionals for Output Functional Estimation of PDEs.
- Source Reconstruction: Inverse problems, such as identifying an unknown source within a physical system, often focus on single-variable dependencies and fail to capture the complexity of 3D domains, as discussed by Zhi Li et al. in The Tikhonov Regularization Method in Hilbert Scales for Determining the Unknown Source for the Modified Helmholtz Equation.
- Environment Invariance: Isolating intrinsic physical laws from environmental noise across multiple environments is an open challenge; Shurui Gui et al. note in Discovering Physics Laws of Dynamical Systems via Invariant Function Learning that generalizing invariant learning from ODEs to PDEs requires addressing complex multi-scale and multi-resolution issues not present in simpler systems.
- Digital Twins: Building interpretable digital twins that discover new physics terms online works well for ODEs, but scaling these sparse discovery frameworks to spatially varying phenomena like fluid dynamics remains an open challenge, as detailed by Tapas Tripura et al. in Probabilistic machine learning based predictive and interpretable digital twin for dynamical systems.
Emerging Paradigms for Spatial Scalability
To break through these dimensionality barriers, researchers are moving away from treating space as a collection of disjointed time-series or a rigid, high-resolution grid. Instead, they are designing architectures that natively incorporate spatial features and continuous operators.
1. Spatial Control Terms
Rather than letting a neural network struggle to learn spatial interactions implicitly, researchers can inject spatial features directly into the continuous-depth architecture. In ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence, Wenjie Mei et al. demonstrate that by incorporating specialized control terms to handle spatial features, neural differential equations can effectively model complex systems governed by PDEs (such as fluid flow). This approach maintains predictive accuracy and stability even when the spatial scale of the system increases, showing high robustness to domain expansion.
2. Continuous Solution Operators
Another major paradigm shift is moving from learning specific trajectories to learning solution operators. Traditional neural differential equations require computationally expensive iterative solvers or fixed time grids. By contrast, operator learning frameworks—such as the Continuous-Time Fourier Neural Operator (CTFNO) introduced by the authors in Learning PDE Solution Operator for Continuous Modeling of Time-Series—learn a continuous mapping from entire function spaces to function spaces. This allows the model to predict system states at any arbitrary spatio-temporal coordinate in a single forward pass, bypassing the sequential bottlenecks of standard ODE solvers.
3. Parallel-in-Time Integration
To handle the sheer volume of data generated by high-resolution continuous fields, researchers are also rethinking how networks are trained. Stefano Massaroli introduces Differentiable Multiple Shooting Layers (MSLs) in Differentiable Multiple Shooting Layers, breaking the sequential bottleneck of continuous-depth models by solving differential equations in parallel across chunks of time. While originally applied to discretized systems, adapting these parallel integration techniques to direct, high-resolution continuous field modeling represents a vital path forward for handling the combined complexity of high-resolution spatial and temporal dimensions.
By shifting the focus from point-wise tracking to continuous, structure-preserving spatial operators, the field is steadily overcoming the scaling limitations of early physics-informed models, paving the way for data-driven discovery in full 3D physical environments.
Key Foundational Work
Accelerating Bayesian Inference over Nonlinear Differential Equations with Gaussian Processes (2008) — contributed 1 concept elements to this direction.
Preserving multiple first integrals by discrete gradients (2011) — contributed 1 concept elements to this direction.