Deep learning has traditionally excelled at mapping inputs to outputs by optimizing scalar losses via first-order gradients ($\nabla_\theta \mathcal{L}$). However, Scientific Machine Learning (SciML)—including Physics-Informed Neural Networks (PINNs) and neural operator learning—demands that models obey physical laws expressed as Partial Differential Equations (PDEs). Enforcing these laws requires computing high-order derivatives of the network's outputs with respect to its inputs (such as Laplacians, Hessians, or biharmonic operators).
While standard Automatic Differentiation (AD) frameworks handle first-order backpropagation seamlessly, they hit a severe performance bottleneck when forced to compute high-order derivatives. A growing body of research focuses on diagnosing this "automatic differentiation overhead" and developing algorithmic workarounds to scale physics-constrained models.
The Problem: The Nested Backpropagation Memory Wall
Standard AD frameworks like PyTorch or JAX are heavily optimized for reverse-mode differentiation (backpropagation). Reverse-mode AD is highly efficient when computing the gradient of a single scalar output with respect to millions of parameters. However, physical residuals require computing derivatives of outputs with respect to input coordinates (e.g., $\frac{\partial^2 u}{\partial x^2}$).
To compute a second-order or higher derivative, the framework must perform nested backpropagation. It builds a computational graph for the first derivative, and then builds another computational graph on top of that intermediate graph to compute the next derivative level. As noted by Dangel et al. and Shi et al., this architecture triggers a "double curse":
- Exponential Graph Complexity: The computational graph grows exponentially with the order of the derivative.
- Polynomial Tensor Growth: The size of the underlying tensors scales poorly with input dimensionality.
For a deep network, tracking every intermediate activation across multiple nested backward passes quickly exhausts GPU memory. This memory wall prevents researchers from scaling physics-constrained architectures to the size of modern foundation models (Richter-Powell et al.) or utilizing large batch sizes during training (Sharma et al.).
Standard Backprop (First-Order):
Inputs ──> [Forward Pass] ──> Output ──> [Backward Pass] ──> First Derivatives
Nested Backprop (High-Order):
Inputs ──> [Forward Pass] ──> Output ──> [Backward Pass 1] ──> [Backward Pass 2] ──> High-Order Derivatives
└─(Must retain entire graph 1 in memory)─┘
To bypass this bottleneck, researchers have developed four primary strategies: graph collapsing, stochastic estimation, coordinate separability, and hybrid numerical discretization.
Strategy 1: Graph Collapsing and Forward Propagation
Instead of blindly stacking reverse-mode passes, can we optimize the computational graph itself or change the propagation direction?
Dangel et al. introduce Collapsed Taylor Mode, a graph-optimization technique that exploits the linearity of differential operators. Instead of evaluating high-order partial derivatives independently through a massive computational graph, their approach sums derivative coefficients during propagation rather than after. This eliminates redundant operations and slashes memory consumption through automated graph rewrites.
Alternatively, forward-mode AD tracks how inputs perturb intermediate layers, making it highly efficient when the number of network outputs or intermediate terms exceeds the number of inputs. He et al. leverage this insight to develop DOF (Differential Operator with Forward-propagation), a framework that accelerates second-order differential operators by restructuring how derivatives traverse the network, achieving up to 20× speedups without losing analytical precision.
Strategy 2: Stochastic and Identity-Based Estimation
What if we avoid computing exact analytical derivative graphs altogether? If we can accurately estimate the behavior of a high-order operator using only forward passes, we can bypass the nested AD bottleneck entirely.
He et al. propose a method that uses Gaussian smoothing and Stein’s Identity to estimate second-order derivatives. By perturbing inputs randomly and applying statistical identities, they extract curvature information in a single forward pass, eliminating the need for nested computational graphs.
For ultra-high-dimensional problems where even basic tensor operations struggle, Shi et al. introduce the Stochastic Taylor Derivative Estimator (STDE). STDE couples high-order univariate automatic differentiation with randomized directional derivatives. This combination breaks the polynomial scaling curve, allowing researchers to solve high-order PDEs in massive dimensional spaces with minimal memory footprints.
Strategy 3: Coordinate Separability
The memory wall is exacerbated when physical equations operate over multi-dimensional spatiotemporal coordinates (e.g., $x, y, z, t$). Enforcing a physics loss across a dense grid means the AD framework must track cross-derivative interactions across thousands of points simultaneously.
To mitigate this, Cho et al. developed Separable Physics-Informed Neural Networks (SPINN). Instead of passing a combined coordinate vector into a single massive network, SPINN processes each coordinate axis using independent, 1D "trunk" networks. The outputs are later combined via a tensor product.
Standard PINN:
(x, y, z) ──> [ Unified Deep Network ] ──> Output u ──> Massive 3D AD Graph
Separable Architecture (SPINN / SepONet):
x ──> [ 1D Trunk X ] ──┐
y ──> [ 1D Trunk Y ] ──┼─> [ Tensor Product ] ──> Output u ──> Cheap 1D AD Graphs
z ──> [ 1D Trunk Z ] ──┘
This structural change fundamentally alters the derivative math. Because the networks are decoupled by coordinate, computing a spatial derivative simplifies into a series of cheap, independent 1D operations. Yu et al. extended this concept to operator learning with Separable Operator Networks (SepONet), demonstrating that coordinate separability shatters the memory wall, enabling orders-of-magnitude faster training on high-dimensional spatiotemporal problems.
Strategy 4: Hybrid Numerical Discretization
A final paradigm shifts the burden of differentiation away from neural networks entirely, blending deep learning with classical numerical analysis.
Sharma et al. introduce DT-PINNs, which replace automatic differentiation for spatial derivatives with high-order meshless numerical discretizations, specifically Radial Basis Function-Generated Finite Differences (RBF-FD). The neural network is only used to evaluate the solution values at specific points, while its spatial derivatives are computed using highly optimized, static numerical stencils.
Similarly, the authors of Mollifier Layers introduce specialized convolutional layers that act as noise-robust differentiators. By replacing analytical AD with localized convolutional operations, these hybrid methods eliminate graph-tracking overhead entirely, providing a stable, fast, and memory-efficient pathway for training deep models on complex physical systems.