Training a neural network to respect the laws of physics is often a tug-of-war. On one side, the network tries to minimize empirical data loss (fitting the observations); on the other, it tries to minimize "model loss" (satisfying differential equations). When these two forces are applied simultaneously from the start, the optimization landscape often becomes treacherous, leading to unstable convergence or physically impossible solutions.
To solve this, researchers have converged on a shared strategy: staged training. Instead of solving the full problem at once, the optimization is broken into a sequence of phases that gradually introduce complexity, ensuring the model is "warmed up" before tackling the most rigid physical constraints.
The Problem of Simultaneous Optimization
In standard Physics-Informed Neural Networks (PINNs), the loss function is a weighted sum of data residuals and physics residuals. However, Zhongwang Zhang, Zhi-Qin John Xu, and Lulu Zhang have identified a "loss-jump" phenomenon in Loss Jump During Loss Switch in Solving PDEs with Neural Networks. They found that when a model transitions from data-driven learning to physics-based constraints, its accuracy can suddenly plummet. This happens because data loss and physics loss often have different "frequency preferences," causing the network to lose its grip on the underlying patterns the moment the physics terms are introduced.
Staged training mitigates this by guiding the optimizer through a curriculum, preventing it from becoming trapped in local minima early in the process.
Anchoring in Data Before Enforcing Physics
A common staged approach involves establishing a baseline fit using empirical data before introducing the governing equations. In Physics-data combined machine learning for parametric reduced-order modelling, Jinlong Fu and colleagues use a sequential procedure that initializes the model with empirical data. This "anchors" the model in the correct region of the solution space. Only then are physics-based constraints introduced via finite differences to refine the model’s behavior, especially in regions where data is sparse.
Similarly, in oceanography, Zhixi Xiong and the team behind MDRF-Net first train on raw observational data to establish a reasonable initial state. This avoids the computational expense of calculating complex partial derivatives until the model has already converged to a plausible solution.
Hierarchical Constraint Enforcement
Not all physical laws are created equal. Some, like boundary conditions or conservation laws, are fundamental "hard" requirements that define the search space, while others, like complex momentum flux, describe the fine details of the dynamics.
Jian Du and Qi Liao demonstrate this in A Knowledge-Inspired Hierarchical Physics-Informed Neural Network. Their strategy narrows the optimization search space by introducing constraints sequentially:
- Boundary Conditions: Where the system meets its limits.
- Initial Conditions: Where the system starts.
- Governing Equations: How the system evolves.
This hierarchy is also visible in fluid dynamics. Weizheng Zhang and Qiang Du show in MUSA-PINN that prioritizing mass conservation before introducing momentum flux-balance terms significantly improves stability. By establishing a physically valid flow field first, the model avoids the instabilities that arise when trying to solve coupled, non-linear equations from scratch.
Penalty Scheduling and Homotopy
Rather than discrete stages, some researchers use a continuous "warm-up" for physics. This is often implemented as a penalty-based schedule, where the weight of the physics loss term starts at zero and increases over time.
In DAE-PINN, Christian Moya and Guang Lin use this to solve differential-algebraic equations. By gradually increasing the penalty for violating algebraic constraints, they ensure the model satisfies hard requirements without getting stuck in local minima early on. Johannes Fischer uses a similar exponential scheduling in PITA to ensure vehicle trajectories remain smooth and realistic.
A more sophisticated version of this is the homotopy continuation strategy used in HomPINNs. Here, Haoyang Zheng and Wenrui Hao use a tracking parameter to transition the objective from a physics-constrained loss to a data-driven loss. This allows the model to follow a continuous path from a known physical solution to one that matches empirical observations.
Decoupling Physics from Neural Augmentation
In "grey-box" modeling, the goal is to use a neural network to fix the errors in a known physical model. A major risk is that the neural network might "over-explain" the data, compensating for incorrect physical parameters (like mass or friction) rather than learning the missing dynamics.
To prevent this, András Retzler proposes a two-stage optimization for industrial robots. First, the physical parameters of the analytical model are calibrated. Only after these are fixed is the neural network trained to handle the remaining residuals. This ensures the physical model remains the primary driver, enhancing the interpretability and generalization of the final system.
Why It Works: Narrowing the Search Space
The intuition behind all these methods is the same: high-dimensional optimization is easier when the search space is constrained. By using data to find the right "neighborhood" or using simple physics to find the right "manifold," staged training ensures that when the full, complex physical equations are finally applied, the neural network is already close enough to the true solution to converge successfully.