Ill-Conditioned Loss Landscapes in Physics-Informed Neural Networks

94 concepts69 papers1 proposal

Physics-Informed Neural Networks (PINNs) offer an elegant promise: solve partial differential equations (PDEs) by embedding physical laws directly into a neural network's loss function as soft regularizers. In theory, a sufficiently deep network can approximate any complex physical system. In practice, however, training these networks frequently fails, resulting in trivial or non-physical solutions.

As Aditi S. Krishnapriyan et al. demonstrated in Characterizing possible failure modes in physics-informed neural networks, these failures are rarely caused by a lack of network expressivity. Instead, they stem from a fundamental optimization bottleneck: embedding differential operators transforms what should be a smooth optimization surface into a highly non-convex, jagged, and ill-conditioned loss landscape.

Why Differential Operators Distort the Parameter Space

To understand why physics-informed training is uniquely difficult, consider standard supervised deep learning. In a supervised setting, the loss function measures the pointwise discrepancy between the network's predictions and the true labels. The optimization landscape is determined entirely by the network architecture and the data distribution.

In a PINN, the loss function forces the network's derivatives to satisfy a differential equation. For a hidden state or output $u(x)$, the network must minimize a residual containing terms like $\nabla u$ or $\Delta u$. As Chuizheng Meng et al. note in When physics meets machine learning: a survey of physics-informed machine learning, evaluating these high-order derivatives via automatic differentiation introduces extreme non-convexity.

When we map the natural "valley" of valid physical solutions back to the network's parameter space (its weights and biases), the landscape becomes severely distorted. A tiny perturbation in a single weight can cause massive, unpredictable oscillations in the higher-order derivatives of the output. In nonlinear regimes, this structural complexity is magnified. Andrea Bonfanti et al. showed in The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks that the Hessian of the residuals does not vanish even as network width increases. This persistent curvature creates a terrain filled with sharp, narrow local minima and steep, unnavigable walls.

The severity of this ruggedness is not arbitrary; it is directly tied to the mathematical properties of the physical system being modeled. In Preconditioning for Physics-Informed Neural Networks, Chang Su establishes that the condition number of the underlying PDE operator dictates the stability of the neural network's loss landscape. If a physical system is analytically ill-conditioned—such as fluid dynamics at high Reynolds numbers or chaotic systems—the neural network inherits this ill-conditioning in its parameter space.

This inheritance manifests as extreme spectral stiffness. In Challenges in Training PINNs: A Loss Landscape Perspective, Pratik Rathore et al. isolate the root cause of this ill-conditioning, proving it is driven primarily by the differential operators in the residual term rather than the boundary or initial conditions. The curvature of the resulting optimization landscape becomes wildly inconsistent across different parameter directions. Standard optimizers find themselves trapped in a terrain composed of steep, narrow canyons and vast, flat plateaus. First-order methods like Adam oscillate violently across the canyon walls without making forward progress, while second-order methods like L-BFGS stall on the flat plateaus.

The Scalability Paradox

In traditional machine learning, adding more training data generally improves the optimization trajectory by smoothing out empirical risk fluctuations. However, physics-constrained learning suffers from a counter-intuitive scalability paradox.

Pratik Rathore et al. discovered that the ill-conditioning of physics-informed loss landscapes actually worsens as the density of collocation points increases. As you sample the physical domain more densely to achieve higher resolution, the differential constraints at adjacent points become highly correlated. This hyper-density compresses the eigenvalues of the optimization Hessian, making the landscape increasingly irregular and difficult to navigate. Consequently, trying to solve a PDE at a higher resolution makes the optimization problem exponentially harder to solve.

Gradient Flow Pathologies

This geometric distortion leads directly to what Sifan Wang et al. term "gradient flow pathologies" in Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks. Because the total loss is a composite of boundary conditions, initial conditions, and PDE residuals, the gradients originating from these different terms often conflict.

In a distorted parameter space, the gradient vectors from the boundary conditions and the physical residuals can point in near-opposite directions. As a result, they cancel each other out or drive the network toward trivial shortcuts—such as predicting a constant zero solution—that satisfy the boundary conditions perfectly but violate the underlying physics entirely. Yichen Luo et al. expand on this in Mitigating Gradient Pathology in PINNs through Aligned Constraint, noting that without full constraints, the inherent non-uniqueness of partial PDE solutions maps to a highly unstable optimization environment where models easily stall.

Strategies for Flattening the Landscape

To bypass these optimization bottlenecks, researchers have developed methods designed to smooth, precondition, or systematically deform the loss landscape:

  • Homotopy and Curriculums: Instead of forcing a network to solve a stiff, highly non-linear equation from a random initialization, authors have proposed starting with an easily optimizable, smooth system and gradually introducing the non-linearities. For instance, Chuqi CHEN et al. introduce this concept in Learn Singularly Perturbed Solutions via Homotopy Dynamics, continuously deforming an easy solution path toward a complex target to avoid local minima.
  • Neural Preconditioning: Borrowing from classical numerical analysis, recent frameworks use learned neural preconditioners to modulate gradient updates. As explored by Lise Le Boudec et al. in Apprentissage d'algorithmes itératifs pour résoudre des EDPs, these preconditioners dynamically scale updates based on the current residual, effectively flattening the stiff directions of the loss surface.
  • Advanced Geometry Optimizers: Rather than relying on standard Euclidean gradient descent, Xianliang Xu et al. demonstrate in Fast Convergence of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks that Natural Gradient Descent (NGD) can achieve convergence rates that are entirely independent of the Gram matrix's condition number, allowing the optimizer to cut straight through the ill-conditioned valleys of the parameter space.

Ultimately, understanding PINNs requires shifting our perspective from model capacity to landscape geometry. The challenge of scientific machine learning is not teaching networks the laws of physics, but reshaping the optimization terrain so they can successfully find them.

Key Foundational Work

Characterizing possible failure modes in physics-informed neural networks (2021) — contributed 2 concept elements to this direction.
Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks (2021) — contributed 2 concept elements to this direction.
NH-PINN: Neural homogenization based physics-informed neural network for multiscale problems (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimization_landscape_complexity_5a923bcb,
  title = {optimization landscape complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimization-landscape-complexity/rdr_5a923bcb4ed0437992fbef550f6fcd6f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimization_landscape_complexity_5a923bcb_v1,
  title = {optimization landscape complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimization-landscape-complexity/rdr_5a923bcb4ed0437992fbef550f6fcd6f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake