In the theoretical landscape of deep learning, neural networks are universal approximators. In practice, however, when applied to scientific computing and partial differential equations (PDEs), they hit a stubborn wall.
While traditional numerical methods like the Finite Element Method (FEM) or spectral methods routinely achieve high-fidelity results with error tolerances of $10^{-6}$ or lower, standard neural network solvers frequently stall. Whether training a Physics-Informed Neural Network (PINN) or a Fourier Neural Operator (FNO), relative errors often plateau between $1\%$ and $0.1\%$ ($10^{-2}$ to $10^{-3}$).
This discrepancy is known as the numerical precision ceiling or accuracy floor. For high-stakes engineering tasks—such as calculating structural stress on critical components or simulating turbulent fluid dynamics—this precision gap makes pure deep learning models insufficient to replace classical simulations.
Relative Error
10^0 +-------------------------------------------------------+
| ### (Standard Neural Network Solvers Stagnate Here) |
10^{-2}| ### |
+-------------------------------------------------------+ <-- The Precision Ceiling
10^{-4}| |
| |
10^{-6}| *** (Traditional Numerical Solvers & Machine Limit) |
+-------------------------------------------------------+
To bridge this gap, researchers are diagnosing why gradient descent stagnates and designing new optimization paradigms, adversarial games, and hybrid architectures to push neural solvers toward machine precision.
Why Standard Training Hits a Wall
The precision ceiling is not a limitation of neural network capacity, but rather a failure of practical optimization and numerical conditioning. As Ben Adcock et al. demonstrate in The gap between theory and practice in function approximation with deep neural networks, standard training regimes encounter a hard accuracy ceiling when approximating smooth mathematical functions. While polynomial-based classical methods converge rapidly, gradient-based optimization of neural networks stagnates at a much higher error threshold.
Two primary mechanisms drive this stagnation:
1. Ill-Conditioned Squared Loss Landscapes
Standard PINNs are trained by minimizing the squared residual of a PDE:
$$\mathcal{L} = \| \mathcal{N}(u) - f \|^2$$
Squaring the differential operator severely worsens the condition number of the optimization problem. As the network begins to fit the low-frequency components of the solution, the gradient updates for high-frequency corrections become vanishingly small or highly erratic. As noted by Johannes Müller et al. in Achieving High Accuracy with PINNs via Energy Natural Gradient Descent, standard first-order optimizers (like Adam) and quasi-Newton methods (like L-BFGS) encounter an accuracy plateau where the relative error stagnates, failing to resolve fine-grained physical details.
2. The Singularity and Machine Precision Limit
When a PDE involves sharp gradients, discontinuities, or singularities, the network struggles fundamentally with floating-point representation limits. In Understanding the difficulty of solving Cauchy problems with PINNs, Bo Zhao et al. prove that to approximate sharp transitions or singularities, the weights of a standard neural network must approach infinity. As these weights grow, they collide with machine precision limits, creating a hard error floor that compute scaling cannot overcome.
Strategies for Breaking the Precision Barrier
To surpass this accuracy floor, recent research focuses on changing how neural networks interact with the underlying differential equations.
Adversarial Formulations (Minimax Games)
Instead of minimizing a squared residual, Competitive Physics-Informed Networks (CPINNs), introduced by Qi Zeng et al. in Competitive Physics Informed Networks and expanded in Competitive Physics Informed Networks, reframe PDE training as a zero-sum game between a solver and a discriminator network.
By avoiding the squared loss function entirely, CPINNs eliminate the inherent numerical ill-conditioning. The discriminator actively forces the solver to correct localized errors, allowing the framework to achieve up to four orders of magnitude higher accuracy (down to $10^{-8}$ relative error), effectively reaching the limits of single-precision floating-point arithmetic. A similar minimax philosophy underpins Friedrichs Learning, proposed by Fan Chen et al. in Friedrichs Learning: Weak Solutions of Partial Differential Equations via Deep Learning, which parameterizes both the solution and test functions as networks to robustly identify weak solutions in complex domains.
Geometry-Aware Optimization
Standard gradient descent updates network parameters based on the geometry of the parameter space ($\mathbb{R}^d$). However, a small step in parameter space can cause unpredictable, non-physical changes in the actual function space.
To fix this, Johannes Müller et al. developed Energy Natural Gradient Descent (E-NGD). By leveraging the Hessian of the PDE's energy functional to define a Riemannian metric, E-NGD performs a Newton-style update directly in the function space. Similarly, Zhuo Chen et al. proposed the Time-Evolving Natural Gradient (TENG) framework in TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision. TENG combines variational principles with natural gradient optimization to bypass standard optimization plateaus, pushing deep learning models toward true machine precision.
Standard Gradient Descent:
[Parameter Step] ----> Unpredictable Function Change (Stalls at 10^-3)
Natural / Energy Gradient Descent:
[Riemannian Metric] --> Precise Step in Function Space (Reaches Machine Precision)
Hybrid Solvers and Preconditioning
Another powerful paradigm acknowledges that while purely data-driven neural operators excel at rapid estimation, they lack the mathematical consistency to refine solutions beyond a certain threshold.
In Neural operators meet conjugate gradients: The FCG-NO method for efficient PDE solving, Alexander Rudikov et al. present a hybrid approach. Instead of using a Neural Operator to output the final solution, they embed a discretization-invariant Neural Operator as a nonlinear preconditioner within a classical Flexible Conjugate Gradient (FCG) framework. The neural operator provides a lightning-fast initial guess and structural guidance, while the classical conjugate gradient steps guarantee mathematical convergence to machine-level precision.
The Path to Physical Fidelity
The push to break the precision ceiling is vital for moving AI from qualitative visualization to quantitative science. As Xiaofeng Dong et al. point out in Physical Benchmarking for AI-Generated Cosmic Web, deep learning emulators that merely "look right" visually often exhibit errors around 5%, failing the sub-percent precision required for scientific discovery.
Whether through adversarial training, natural gradients, or hybrid classical-neural architectures, overcoming the numerical precision gap is the defining frontier for deep learning in the physical sciences.
Key Foundational Work
The gap between theory and practice in function approximation with deep neural networks (2020) — contributed 3 concept elements to this direction.
Friedrichs Learning: Weak Solutions of Partial Differential Equations via Deep Learning (2020) — contributed 1 concept elements to this direction.
The data-driven localized wave solutions of the derivative nonlinear Schrodinger equation by using improved PINN approach (2021) — contributed 1 concept elements to this direction.