Physics-Informed Neural Networks (PINNs) offer a powerful paradigm for solving partial differential equations (PDEs) by embedding physical laws directly into a neural network's loss function. Instead of relying solely on labeled data, a PINN minimizes a composite loss function typically composed of multiple terms: initial conditions, boundary conditions, data-driven points, and the unsupervised PDE residual itself.
While elegant in theory, training a PINN often fails out of the box. This failure stems from a fundamental optimization challenge: the network must simultaneously satisfy multiple, often competing, objectives. When these distinct loss terms are combined naively, their gradients can conflict, cancel out, or vary by orders of magnitude, leading to severe training instability and inaccurate solutions.
The Anatomy of Gradient Pathology
To understand why PINNs fail, we must look at how gradients flow through the network during backpropagation. In a standard multi-task learning setup, different losses share a backbone network. In PINNs, this problem is exacerbated because the objectives represent fundamentally different mathematical entities: point-wise constraints (like boundary conditions) versus differential operators evaluated across a continuous domain (the PDE residual).
As identified by Wang et al., this structural mismatch introduces severe "gradient flow pathologies." The optimization landscape becomes highly stiff—a phenomenon analogous to the numerical stiffness encountered in traditional, non-neural PDE solvers (Sturm et al.).
During gradient descent, three distinct pathologies typically emerge:
- Gradient Dominance and Suppression: Loss terms with larger initial magnitudes or steeper landscapes (often the boundary conditions or data-driven points) dominate the total gradient vector. As a result, the network prioritizes satisfying these terms while completely suppressing the unsupervised PDE residual loss (Niu et al.; Bischof et al.).
- Opposing Gradients: The gradients derived from the physical equations and those from the boundary conditions can point in opposite directions. Luo et al. demonstrated that these conflicting directional signals trap the model in poor local minima, preventing it from finding a solution that satisfies both constraints simultaneously.
- Differential Explosion and Collapse: Unlike standard deep learning where gradients are computed only with respect to the network weights, PINN training relies on internal differentials—the derivatives of the model's output with respect to its input coordinates. Meng et al. showed that as networks grow deeper, these internal Jacobian matrices tend to either vanish or explode, causing severe numerical instability irrespective of the loss weights.
Dynamic Loss Weighting Strategies
The most direct way to mitigate these pathologies is to dynamically adjust the scalar weights ($\lambda_i$) assigned to each loss component throughout the training process. Rather than relying on tedious and often suboptimal manual tuning (Xiang et al.), adaptive weighting algorithms analyze the training dynamics in real time to balance the optimization landscape.
[Loss Components] ---> [Gradient / Loss Statistics] ---> [Adaptive Weighting Engine] ---> [Balanced Gradient Step]
(PDE, BC, Data) (e.g., ReLoBRaLo, Inverse Dirichlet)
Gradient-Based Annealing
One foundational approach, introduced by Wang et al., utilizes a learning rate annealing algorithm. By tracking the backpropagated gradients of the PDE residual and the boundary conditions, the algorithm dynamically scales the weights to ensure that the magnitudes of the gradient contributions from each task are roughly equalized. Similarly, Sturm et al. proposed inverse-Dirichlet weighting, which estimates the variance of the task-specific gradients to prevent vanishing signals, making the network highly robust when simulating complex, multi-scale physics.
Preventing Over-Correction with Upper Bounds
While adaptive weighting helps, unconstrained weight adaptation can introduce its own failure modes. If a specific loss term becomes temporarily easy to optimize, an adaptive algorithm might aggressively down-weight it, causing the network to forget that constraint entirely. To counter this, Niu et al. introduced an adaptive weighting strategy that incorporates explicit upper bounds. This ensures that even if a dominant term tries to suppress the unsupervised residual loss, the weights remain bounded, preserving a balanced optimization process.
Overcoming Computational Overhead
A major drawback of early gradient-matching methods is their computational cost: calculating explicit gradient statistics for every single loss term requires multiple backward passes per optimization step. To make adaptive balancing practical for large-scale applications, Bischof et al. developed ReLoBRaLo (Relative Loss Balancing with Random Lookback). Instead of computing expensive per-step gradient matrices, ReLoBRaLo evaluates the relative efficiency of loss reduction over time using a moving average and a random lookback mechanism, providing robust stabilization at a fraction of the computational cost.
Structural and Architectural Interventions
Beyond modifying the loss weights, researchers have found that changing the network's architecture or altering how constraints are introduced can fundamentally eliminate gradient conflicts.
Constraint Realignment and Scheduling
Rather than forcing the network to resolve conflicting boundary and physics gradients simultaneously from the start of training, Luo et al. proposed CAML (Constraint Aligned Multi-Task Learning). This method aligns the constraint trajectories and strategically delays the introduction of complex physics residuals. By allowing the network to first master the boundary conditions before introducing the full physical laws, the model avoids the destructive gradient interference that characterizes early-stage PINN training.
Hard Constraints and Jacobian Regulation
Another way to bypass the loss-balancing act is to enforce boundary conditions by construction through the network architecture itself, removing them from the loss function entirely. Deng et al. combined this hard-constraint approach with Fourier-based subnetworks, allowing the model to resolve high-frequency advection-diffusion behaviors without experiencing boundary-vs-residual gradient imbalances.
To address the deep internal derivative failures identified in deep architectures, Meng et al. designed CoupledNet. By incorporating specialized coupling layers, CoupledNet strictly regulates the spectral norms of the input-output Jacobian matrices. This structural constraint prevents internal differentials from exploding or vanishing, enabling PINNs to scale to much greater depths successfully.
Alternative Optimization Formulations
Finally, the problem can be reframed entirely outside of standard multi-objective gradient descent:
- Multi-Task Optimization (MTO): Wang et al. reframed PINN training as an explicit multi-task optimization problem, using auxiliary tasks to enable adaptive knowledge transfer, which proves particularly effective when data points are scarce (Mathias et al.).
- Causal Message Passing: Cho et al. provided a rigorous theoretical foundation for adaptive weighting by incorporating spatial and temporal causality. Their method, MP-PINN, utilizes the Belief Propagation algorithm to pass messages across the spatio-temporal domain, ensuring that loss weights respect physical causality (e.g., ensuring the past informs the future).
- Parameter Rescaling: For problems involving disparate physical constants, Chi et al. demonstrated that a fundamental rescaling of the penalty parameters based on the underlying differential operators is essential to guarantee stable training and spectral accuracy.
By treating the training of physics-informed models not as a single rigid objective, but as a dynamic system of interacting gradients, these adaptive weighting and structural strategies transform PINNs from fragile numerical experiments into robust, scalable solvers for complex physical phenomena.
Key Foundational Work
Self-adaptive loss balanced Physics-informed neural networks (2022) — contributed 4 concept elements to this direction.
Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks (2021) — contributed 2 concept elements to this direction.
Augmenting a Physics-Informed Neural Network for the 2D Burgers Equation by Addition of Solution Data Points (2022) — contributed 1 concept elements to this direction.