Physics-Informed Neural Networks (PINNs) offer a promising path toward mesh-free solutions of partial differential equations (PDEs) by embedding physical laws directly into the loss function. However, as noted by Lu Lu et al., these models are notoriously difficult to train. The primary obstacle is not just the non-convexity of the loss landscape, but a phenomenon often described as gradient pathology or gradient interference.
In a standard PINN, the total loss is a weighted sum of multiple competing objectives: the residual of the governing PDE, the boundary conditions (BCs), and the initial conditions (ICs). When these terms are simply added together, the optimization process often becomes a "tug-of-war" where satisfying one physical constraint directly impedes the satisfaction of another.
The Pathology of Magnitude Imbalance
The most immediate challenge in training physics-constrained models is the discrepancy in the scales of different loss components. As Yao et al. explain, gradients often suffer from pathologies where updates are dominated by the most numerically significant loss term. For example, if the PDE residual produces gradients orders of magnitude larger than the boundary condition loss, the optimizer will prioritize the physics of the domain while effectively ignoring the boundaries. This leads to solutions that may look physically plausible in isolation but are globally incorrect or impossible.
This numerical stiffness is exacerbated by the fact that different components converge at unequal speeds. Wang et al. characterize this as a fundamental stiffness in the optimization landscape, where the model's bias toward learning functions with larger magnitudes prevents it from simultaneously satisfying all physical constraints.
Directional Conflict and Gradient Interference
While magnitude imbalance is a significant hurdle, Wang et al. argue that directional opposition between gradient vectors is actually more detrimental to convergence. Even if two loss terms have similar magnitudes, their gradients might point in nearly opposite directions. In such cases, a step taken to minimize the PDE residual might significantly increase the boundary error.
Hwang et al. point out that standard optimization algorithms typically converge to a stationary point of the total loss sum rather than finding a true Pareto optimal balance. This lack of Pareto stationarity means the model can "get stuck" in a state where the total loss appears low, yet essential physical properties remain unsatisfied. Elhamod et al. further observe that these conflicts can trap models in non-physical local minima, such as higher-energy states in eigenvalue problems that are mathematically valid but physically irrelevant.
Geometric and Adaptive Solutions
To resolve these conflicts, researchers have moved away from simple manual weighting—which Perdikaris et al. and Yang describe as a tedious and often unsuccessful trial-and-error process—toward automated, geometrically-aware strategies.
1. Gradient Projection and Cones
One sophisticated approach involves treating the various loss components as geometric constraints. Bu et al. propose identifying a "Harmonized Cone"—a shared feasible region in the gradient space. By restricting optimization updates to the intersection of the primal and dual gradient cones, the model avoids "dead zones" where gradients for different constraints counteract each other. Similarly, Peng et al. utilize gradient projection to stabilize training in 3D domains, preventing oscillations between boundary fitting and internal smoothness.
2. Adaptive Weighting
Another strategy involves dynamically adjusting the importance of each loss term during training. Sturm et al. introduced "inverse-Dirichlet weighting" to prevent vanishing task-specific gradients, ensuring that no single objective is neglected. Other methods, such as the Neural Tangent Kernel (NTK)-guided training proposed by Wang et al., use the network's own architecture to balance the convergence rates of different terms.
3. Reformulating the Problem
Some researchers suggest that the "soft constraint" approach (adding penalties to the loss) is fundamentally limited. Négiar et al. propose "hard constraints" through differentiable solvers, while Zhou et al. use a saddle-point optimization framework to narrow the gap between competing gradient magnitudes. By decoupling the enforcement of boundary conditions from the primary physics solver, as seen in the work of Zhou et al., the model can bypass the gradient conflict entirely.
Ultimately, the goal of this research direction is to transform PINN training from a fragile balancing act into a robust numerical method. As Zhang et al. demonstrate through comprehensive benchmarking, resolving these gradient pathologies is the key to making neural solvers competitive with traditional numerical methods across diverse and complex physical systems.
Key Foundational Work
DeepXDE: A Deep Learning Library for Solving Differential Equations (2020) — contributed 1 concept elements to this direction.
Adversarial uncertainty quantification in physics-informed neural networks (2019) — contributed 1 concept elements to this direction.