Physics-Informed Neural Networks (PINNs) have emerged as a powerful paradigm for solving partial differential equations (PDEs) by embedding physical laws directly into the loss function of a neural network. In a standard setup, the network minimizes a residual loss—the degree to which the network's output violates the governing differential equations and boundary conditions. Intuitively, if the residual loss approaches zero, the network should converge to the true physical solution.
However, a growing body of theoretical and empirical evidence reveals a fundamental flaw in this assumption: the residual-accuracy gap. Minimizing the residual loss does not inherently guarantee a corresponding reduction in the actual solution error relative to the ground truth physics. In fact, PINNs frequently experience "silent failures," where the training loss drops to near-zero while the model predicts a completely incorrect or physically meaningless state.
The Trivial Solution Trap
The most glaring manifestation of the residual-accuracy gap is the tendency of PINNs to collapse into trivial solutions. Consider a homogeneous PDE or a stochastic system like the Fokker-Planck equation without constant terms. If a neural network outputs zero everywhere ($u(\mathbf{x}) = 0$), its derivatives are also zero. Consequently, the differential operator evaluates to zero, perfectly satisfying the interior PDE residual.
As Hao Zhang et al. demonstrate in Solving Fokker-Planck equation using deep learning and Arka Daw et al. show in Mitigating Propagation Failures in Physics-informed Neural Networks using Retain-Resample-Release (R3) Sampling, this "zero-solution trap" acts as a powerful mathematical local minimum. Because a flat zero output is incredibly simple for a neural network to learn, the optimizer often prioritizes this trivial fit over the complex, highly oscillatory dynamics of the true physical system.
Michael Penwarden et al. note in A unified scalable framework for causal sweeping strategies for Physics-Informed Neural Networks (PINNs) and their temporal decompositions that this failure mode is particularly acute in time-dependent problems. When a network fails to respect the temporal flow of information, it often satisfies the differential residual globally by collapsing to zero and ignoring the initial conditions entirely.
The Conflict of Objectives: Soft Constraints and Ill-Conditioning
Why does the optimizer ignore boundary or initial conditions in favor of a trivial or incorrect state? The answer lies in how these constraints are traditionally enforced. Most PINN frameworks treat boundary conditions as soft penalty terms within a composite loss function:
$$\mathcal{L} = \lambda_{pde} \mathcal{L}_{pde} + \lambda_{bc} \mathcal{L}_{bc}$$
This formulation turns training into a multi-objective optimization problem. As Yiqi Gu et al. argue in Structure Probing Neural Network Deflation and Chenhong Zhou et al. show in Dual-Balancing for Physics-Informed Neural Networks, this creates an incredibly difficult optimization landscape. When the optimization process prioritizes the general physical laws ($\mathcal{L}_{pde}$) over specific boundary conditions ($\mathcal{L}_{bc}$), the network may converge to a solution that is physically plausible but mathematically incorrect because it is not properly anchored to the unique constraints of the problem instance. Atiyo Ghosh et al. highlight this exact decoupling in Harmonic Neural Networks, showing that models can successfully minimize physics-based loss while completely failing to capture boundary states.
Standard PINN Loss Landscape (Soft Constraints)
Loss
│ ▲
│ ╱ ╲ ┌─────────────────────────────────┐
│ ╱ ╲ │ Trivial Local Minimum │
│ ╱ ╲ │ (Low PDE Loss, High True Error) │
└───v───────v──┴─────────────────────────────────┘
Zero True
Solution Solution
Furthermore, this optimization problem is often severely ill-conditioned. The authors of Preconditioning for Physics-Informed Neural Networks show that in ill-conditioned physical systems, a low training loss does not guarantee high prediction accuracy because the residual norm does not scale proportionally with the true error norm. This instability is exacerbated in high-dimensional non-linear PDEs. For instance, Chuwei Wang et al. prove in Is $L^2$ Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? that the standard $L^2$ loss function lacks stability in high dimensions, allowing small training residuals to coexist with massive deviations from the ground truth.
Silent Failures and Representability Limits
Even when the optimizer behaves perfectly, the residual-accuracy gap can emerge from a mismatch between the network's mathematical capacity and the true solution's properties. Yuqian Cheng et al. establish in Consistency of Physics-Informed Neural Networks for Second-Order Elliptic Equations that if a target function possesses derivative discontinuities or other non-smooth properties that place it outside the network's representable space, the system can experience a silent failure. The network confidently stabilizes at an incorrect solution while the training loss continues to decrease, masking the underlying failure.
This divergence between loss and actual error breaks downstream tasks like Neural Architecture Search (NAS). Bo Zhang et al. point out in Discovering physics-informed neural networks model for solving partial differential equations through evolutionary computation that using physics-informed loss functions as fitness indicators in evolutionary computation does not correlate perfectly with actual model error. Automated search algorithms can inadvertently select poorly performing architectures that are merely adept at minimizing the residual without capturing the true physics.
Escaping the Trap: Architectural and Algorithmic Solutions
To make neural PDE solvers reliable, researchers have developed mechanisms to explicitly bridge or bypass the residual-accuracy gap.
1. Hard Constraints and Exact Boundaries
Instead of balancing competing objectives via penalty terms, one can force the network architecture to satisfy boundary conditions exactly. Johannes Müller et al. mathematically prove in Notes on Exact Boundary Values in Residual Minimisation that enforcing boundary conditions exactly within the network architecture is the only way to achieve strong $H^2$ error estimates, turning the training loss into an "honest" indicator of the true solution's error.
2. Decoupling and Alternative Objectives
To prevent the zero-solution collapse in probability density equations, Feng Liu et al. propose in Score-based free-form architectures for high-dimensional Fokker-Planck equations decoupling the learning of a probability density's shape from its normalization using a "score PDE loss," ensuring the model cannot collapse into a trivial zero state.
To handle multi-solution systems and push networks away from simple, trivial local minima, Yiqi Gu et al. introduce deflation operators in Structure Probing Neural Network Deflation that systematically penalize known solutions, forcing the network to discover complex, oscillatory states. Similarly, Alexander Ilin et al. leverage model ensembles in Improved Training of Physics-Informed Neural Networks with Model Ensembles, using the disagreement between ensemble members to safely expand the training domain from known data points without falling into trivial traps.
3. Rigorous Error Certification
Finally, to address the lack of mathematical guarantees, recent work focuses on provable error bounds. Shuheng Liu 0001 et al. introduce in Residual-based error bound for physics-informed neural networks a suite of algorithms that calculate provable error bounds by analyzing the equation's structure alongside the network's residual.
Taking this a step further, Francisco Eiras et al. developed $\partial$-CROWN in Efficient Error Certification for Physics-Informed Neural Networks, a framework providing formal worst-case error guarantees across the entire continuous domain by extending neural network verification techniques to partial derivatives, transforming PINNs from black-box optimizers into certified scientific tools.
Key Foundational Work
Notes on Exact Boundary Values in Residual Minimisation (2022) — contributed 1 concept elements to this direction.
Improved Training of Physics-Informed Neural Networks with Model Ensembles (2022) — contributed 1 concept elements to this direction.
Is $L^2$ Physics Informed Loss Always Suitable for Training Physics Informed Neural Network? (2022) — contributed 1 concept elements to this direction.