Physics-Informed Neural Networks (PINNs) have revolutionized scientific computing by allowing us to solve partial differential equations (PDEs) without traditional meshes. However, they face a fundamental "memory wall." To enforce physical laws, these models must compute high-order derivatives of the network's output with respect to its inputs.
In standard deep learning frameworks, this is done via Automatic Differentiation (AD). While AD is elegant, it is computationally "blind" to the mathematical structure of the physics it is trying to represent. As the order of the PDE increases—moving from simple gradients to Laplacians or biharmonic operators—the computational graph required by AD explodes in size, leading to massive memory overhead and sluggish training.
Researchers are now developing a suite of techniques to bypass this bottleneck, replacing generic AD with more efficient, "physics-aware" ways of calculating derivatives.
The Memory Wall of Automatic Differentiation
To understand why PINNs struggle, consider a second-order PDE like the Fokker-Planck equation. To calculate the loss, the network must compute the second derivative of its output. In a standard AD system, this requires "nesting" the differentiation: the system builds a graph to compute the first derivative, and then builds another graph on top of that to compute the second.
As noted by Marlon Sproesser Mathias and colleagues in their work on the 2D Burgers Equation, this process is significantly more memory-intensive than standard supervised learning. This overhead creates a practical bottleneck where the number of physics evaluation points is limited by available hardware, often preventing models from scaling to 3D simulations or high-resolution grids.
Bypassing the Graph with Forward Propagation
One promising direction is to rethink how derivatives flow through the network. Instead of building a massive backpropagation graph, researchers like Di He have developed DOF (Differential Operator with Forward-propagation).
The intuition here is to propagate the derivative information forward alongside the network's activations. By using a one-step derivative rule, DOF can calculate complex second-order operators up to 20x faster than standard methods. Similarly, Jihun Han and Mihai Nica proposed a derivative-free method for elliptic PDEs, proving that we can solve these equations without ever calculating the derivatives of the network itself.
Shifting Domains: Spectral and Integral Methods
If calculating derivatives in spatial coordinates is hard, why not move to a domain where differentiation is easy?
Spectral Methods take advantage of the fact that in the frequency domain, differentiation is simply multiplication. Tianchi Yu and Ivan Oseledets introduced Spectral Informed Neural Networks (SINNs), which shift the problem to the frequency domain using Fourier transforms. This eliminates the need for spatial AD entirely, resulting in a model that is both faster and more accurate. To handle non-periodic boundaries—a classic weakness of Fourier methods—Haydn Maust and Anima Anandkumar developed FC-PINO, which uses Fourier Continuation to suppress oscillations at the edges of the domain.
Alternatively, the Deep Finite Volume Method (DFVM), proposed by Cen Jianhuan and colleagues, uses the divergence theorem to transform second-order derivatives into first-order boundary integrals. By enforcing local conservation laws rather than "strong-form" residuals, DFVM achieves a 6x speedup by reducing the mathematical "order" of the problem the network has to solve.
Hard-Coding Physics into the Architecture
Perhaps the most efficient way to handle derivatives is to ensure the network's output satisfies the PDE by construction, removing the need for a physics loss altogether.
Johannes Hendriks and Thomas B. Schön demonstrated this with Linearly Constrained Neural Networks. By modeling a vector field as the gradient of an underlying "potential" field, they guarantee that the prediction is curl-free or divergence-free without needing to compute those derivatives during training.
Similarly, Xinling Yu introduced Separable Operator Networks (SepONet), which use independent "trunk" networks for each coordinate axis. This "divide and conquer" approach allows for an 82x reduction in GPU memory, enabling the modeling of high-dimensional systems that were previously computationally impossible.
The Path Toward Differentiable Programming
As these methods mature, the field is moving toward a unified framework of Differentiable Programming (DP). As Alan Edelman and Mike Innes argue in their work on Zygote.jl, the goal is to make differentiation a first-class feature of programming languages.
By combining specialized mathematical rules—like Stein’s Identity for stochastic gradients or compile-time symbolic differentiation—with deep learning, we can bridge the gap between traditional numerical analysis and modern AI. The result is a new generation of solvers that are not just "physics-informed," but physics-optimized.