In recent years, physics-informed machine learning has emerged as a powerful paradigm for solving complex physical systems. By embedding differential equations directly into neural networks and probabilistic models, researchers can infer hidden physical states and solve partial differential equations (PDEs) with remarkable empirical success.
However, a profound theoretical gap remains. While these models excel in practice on highly nonlinear systems like fluid turbulence or chaotic dynamics, our rigorous mathematical guarantees—such as convergence proofs, error bounds, and exact constraint enforcement—are almost entirely trapped in a safe haven: the linear operator restriction.
To understand why this restriction exists, we must look at the mathematical shortcuts that linear operators provide, where they break down in the presence of nonlinear physics, and how the field is working to bridge this gap.
The Linear Safe Haven
Why are linear operators so uniquely tractable for physics-informed models? The answer lies in how linearity preserves the underlying mathematical structures of probabilistic and analytical frameworks.
The Preservation of Gaussianity
In Gaussian Process (GP) models, linearity is a superpower. A fundamental property of GPs is that any linear transformation of a Gaussian process yields another Gaussian process. As demonstrated by Maziar Raissi in Machine learning of linear differential equations using Gaussian processes, this allows researchers to treat the parameters of a differential equation as hyperparameters of a GP kernel. Because the joint distribution between the data and the linear differential operator remains Gaussian, physical constants can be inferred exactly from small, noisy datasets.
Similarly, Carl Jidling et al. (Linearly constrained Gaussian processes) and Marc Harkonen et al. (Gaussian Process Priors for Systems of Linear Partial Differential Equations with Constant Coefficients) leverage this property to perform "structural surgery" on covariance functions. They construct kernels whose realizations are mathematically guaranteed to satisfy linear constraints (such as being divergence-free or curl-free). Markus Lange-Hegermann (Linearly Constrained Gaussian Processes with Boundary Conditions) even extends this to exact boundary conditions using symbolic algebra.
Linear Operator: GP Prior ───► [Linear PDE Operator] ───► Still a GP (Exact Inference)
Nonlinear Operator: GP Prior ───► [Nonlinear Operator] ───► Non-Gaussian (Math Breaks)
As L. Mihaela Păun points out in Physics-informed Gaussian Processes for Nonlinear Partial Differential Equations in a Fuid-Dynamics Application, this foundational property completely collapses for nonlinear PDEs. The derivative of a GP under a nonlinear mapping is no longer a GP, making standard Bayesian non-parametric methods difficult to apply directly to complex fluid dynamics.
Analytical Shortcuts and Green's Functions
Linearity also allows classical numerical analysis to provide explicit analytical shortcuts, such as Green's functions or fundamental solutions. For instance, the framework developed by Monika Nagy-Huber et al. (Physics-Informed Boundary Integral Networks (PIBI-Nets): A Data-Driven Approach for Solving Partial Differential Equations) maps boundary integrals directly to neural networks. However, this method is strictly restricted to linear operators with constant coefficients where the fundamental solutions are analytically known, preventing its direct application to nonlinear physics.
Why Optimization and Convergence Proofs Fail
When we transition from linear to nonlinear operators, the optimization landscape of physics-informed models changes from a well-behaved terrain into a highly non-convex, chaotic space.
The Loss of Operator Conditioning
In Physics-Informed Neural Networks (PINNs), training involves minimizing a residual loss. For linear operators, we can reason rigorously about the conditioning of this loss function. Tim De Ryck et al. (An operator preconditioning perspective on training in physics-informed machine learning) proved that the training speed of a PINN is governed by the Hermitian square of the PDE's operator ($D^*D$). By applying preconditioning techniques to this operator, they achieved exponential convergence. Crucially, however, their mathematical proof relies on the operator being linear; extending these conditioning strategies to nonlinear operators remains an open hurdle.
Kernel Instability
Global convergence guarantees for neural network architectures often rely on the stability of the underlying optimization kernels. Vincent Y. F. Tan et al. (On the Convergence of (Stochastic) Gradient Descent for Kolmogorov-Arnold Networks) provided the first rigorous proof of global convergence for Kolmogorov-Arnold Networks (KANs) in physics-informed tasks, but noted that their kernel stability assumptions fail to hold when moving to nonlinear operators.
Without these linear assumptions, proving that a network will successfully minimize residual loss becomes incredibly difficult, a challenge highlighted by Nima Hosseini Dashtbayaz et al. (Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations).
The Complexity Bottleneck
Can neural networks overcome the "curse of dimensionality" when approximating physical fields? For linear equations, the answer is a provable yes. For nonlinear equations, it is still a question mark.
Linear Elliptic/Kolmogorov PDEs ───► Probabilistic/Linear Shortcuts ───► Provable Polynomial Complexity
Nonlinear Dynamics (e.g., Navier-Stokes) ───► No Linear Shortcuts ───► Complexity Bounds Unknown
For linear elliptic equations, Tanya Marwah et al. (Parametric Complexity Bounds for Approximating PDEs with Neural Networks) proved that if the equation's coefficients can be represented by small neural networks, the solution's size grows only polynomially with the input dimension. Similarly, Tim De Ryck et al. (Error analysis for physics-informed neural networks (PINNs) approximating Kolmogorov PDEs) established that PINNs can overcome the curse of dimensionality for Kolmogorov PDEs by exploiting specific linear probabilistic representations.
These mathematical shortcuts do not hold for nonlinear systems like fluid dynamics. This restriction also limits data efficiency guarantees. For example, Unique Subedi et al. (On the Benefits of Active Data Collection in Operator Learning) demonstrated that active data collection can achieve exponential error decay by querying the eigenfunctions of the input distribution's covariance kernel—but their framework is strictly bounded by the assumption of a linear operator.
Strategies for the Nonlinear Frontier
To break past the linear operator restriction, researchers are actively developing new mathematical and architectural paradigms that handle fully nonlinear physics without sacrificing theoretical rigor.
1. Linearization and Perturbation
A common strategy is to approximate nonlinear operators using local linearizations or small perturbation parameters. Mara Daniels et al. (Uncertainty-Aware Diagnostics for Physics-Informed Machine Learning) showed that existing diagnostic tools can be successfully applied to nonlinear PDEs by introducing explicit linearization steps during the diagnostic phase. In the same vein, Shuheng Liu et al. (Residual-based error bound for physics-informed neural networks) developed provable error bounds for nonlinear systems by restricting their analysis to regimes with small perturbation parameters.
2. Neural Operator Splitting
Instead of forcing a neural network to learn a complex nonlinear system all at once, some architectures isolate the nonlinear components from the well-behaved linear evolution components. Li Sun et al. (Origo: Interpretable Multi-physics PDE Foundation Model through Neural Operator Splitting) introduced a foundation model that splits complex multi-physics equations into interpretable "physical atoms." While this allows for strong zero-shot generalization, the authors note that purely nonlinear operators still present an open architectural challenge.
3. Alignment with Classical Iterative Schemes
Rather than treating neural operators as black boxes, a rising direction is to align network layers directly with classical numerical iterations designed for nonlinear problems.
Takashi Furuya et al. (Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equations) bypassed the curse of parametric complexity for nonlinear parabolic equations by mathematically aligning the layers of a neural operator with the steps of a classical Picard iteration. This allowed them to prove that the required model size grows only polynomially as the desired accuracy increases, providing a blueprint for how the field can establish rigorous guarantees for nonlinear physics.
Key Foundational Work
A new maximum principle of elliptic differential equations in divergence form (2008) — contributed 1 concept elements to this direction.
A possible theory for the interaction between convective activities and vortical flows (2011) — contributed 1 concept elements to this direction.