Deep learning has fundamentally changed scientific computing by transforming partial differential equation (PDE) solving into optimization and regression tasks. Models like Physics-Informed Neural Networks (PINNs) and Neural Operators can bypass traditional, computationally expensive grid-based solvers. However, a major architectural divide has emerged: models that excel at finding static, steady-state equilibria often struggle or fail entirely when applied to transient, time-evolving systems.
Extending neural PDE solvers from static snapshots to time-dependent systems requires moving away from fixed-point mappings and toward learning stable physical propagators.
The Steady-State Comfort Zone
To understand why time-dependent problems are challenging, we must first look at why neural networks find steady-state problems relatively straightforward. In a steady-state system (such as an elliptic PDE governing electrostatics or steady-state heat conduction), the target physical field is invariant over time. The neural network's job is to learn a direct mapping from spatial coordinates, boundary geometries, or material parameters to a single, stable equilibrium configuration.
For example, in Deep Equilibrium Based Neural Operators for Steady-State PDEs, Marwah et al. exploit this by treating the PDE solution as a mathematical fixed point, combining Fourier Neural Operators (FNOs) with Deep Equilibrium Models to solve equations efficiently with minimal memory overhead. Similarly, physics-informed convolutional architectures like PhyGeoNet, developed by Wang et al. in PhyGeoNet: Physics-informed geometry-adaptive convolutional neural networks for solving parameterized steady-state PDEs on irregular domain, map irregular spatial boundaries directly to stationary fluid fields.
In these cases, the loss landscape is anchored by static boundary conditions. The network does not need to worry about how the system reached equilibrium, only what the equilibrium looks like.
Why Time Breaks the Static Paradigm
When a physical system evolves over time—such as unsteady fluid flow, wave propagation, or dynamic neutron flux variations—the problem shifts from a static mapping to a trajectory rollout. This transition introduces three major mathematical and computational hurdles:
1. Autoregressive Error Propagation
When neural networks emulate time-dependent PDEs, they typically predict the state of the system at time $t + \Delta t$ given the state at time $t$. If the model's prediction contains even a microscopic error, that error is fed back into the network as the input for the next time step. Over long temporal rollouts, these errors compound exponentially, causing the simulation to drift into unphysical states or blow up entirely.
As Koehler et al. demonstrate in APEBench: A Benchmark for Autoregressive Neural Emulators of PDEs, treating time-dependent physics as a simple image-to-image translation task fails because it ignores the strict requirement for long-term temporal stability.
2. Loss Function Mismatch
Many advanced neural PDE solvers rely on energy-minimizing variational forms rather than evaluating strong-form derivatives. While this works beautifully for elliptic equations, Balu et al. note in NeuFENet: Neural Finite Element Solutions with Theoretical Bounds for Parametric PDEs that these energy functionals are fundamentally optimized for steady states. Adapting them to parabolic or hyperbolic systems requires entirely new variational formulations that can track energy dissipation and conservation across a continuous temporal trajectory.
3. Gradient Pathology
In a standard PINN, the loss function is a weighted sum of boundary conditions, initial conditions, and the PDE residual. When time is introduced as an input coordinate, the gradients originating from the initial conditions often conflict with the gradients from the physics residual deeper in the time domain.
In Mitigating Gradient Pathology in PINNs through Aligned Constraint, Luo et al. show that this "gradient pathology" traps models in poor local minima. Overcoming this requires designing constraint alignment strategies that dynamically adapt as the simulation moves forward in time.
Mechanisms for Temporal Adaptation
To bridge the gap between static snapshots and dynamic trajectories, researchers are developing architectures explicitly designed to capture temporal evolution.
Steady-State Modeling:
[Boundary/Geometry] ───> (Neural Network) ───> [Static Equilibrium Field]
Time-Dependent Modeling:
[Initial State t₀] ───> (Neural Propagator) ───> [State t₁] ───> (Neural Propagator) ───> [State t₂]
▲ ▲
└───────── [Differentiable Physics Solver Feedback] ────┘
Differentiable Physics in the Loop
Rather than training neural networks on isolated, pre-computed datasets, modern frameworks embed differentiable numerical solvers directly into the neural network's training loop. This allows the network to be trained on "unrolled" trajectories, meaning the loss function evaluates the model's accuracy after several steps of interaction with the physics engine. This mechanism forces the neural network to learn representations that remain stable over time, suppressing autoregressive drift.
Transitioning to Linear Propagators
Another approach involves shifting the learning target from the raw physical state to the underlying operator that drives change. In Data-driven discovery of Green’s functions with human-understandable deep learning, Boullé et al. demonstrate how networks can learn fundamental response functions (Green's functions) for steady-state systems. The frontier of this work lies in adapting these architectures into time-dependent linear propagators, effectively uncovering the mathematical "DNA" that dictates how a fluid or wave field updates from one millisecond to the next.
High-Order and Product-Structured Activations
Transient physical phenomena, such as shock waves or turbulent eddies, feature sharp, highly localized changes in space and time. Standard activation functions like ReLU often smooth out these features. To capture high-frequency temporal transitions, Maharaj et al. introduce learnable, product-structured activations in Deep Learning with Learnable Product-Structured Activations. These structures allow individual neurons to capture complex, high-order interactions, providing the mathematical expressivity needed to track transient behaviors without requiring prohibitively deep networks.
Key Foundational Work
A new maximum principle of elliptic differential equations in divergence form (2008) — contributed 1 concept elements to this direction.