Abstract
The "loss-jump" phenomenon in physics-informed neural networks (PINNs) occurs when transitioning training from data-based supervision to physics-based constraints (PDE residuals). This transition often leads to a significant, sudden decrease in model accuracy. This study investigates the underlying causes, identifying conflicting spectral biases between data loss and model loss as the primary driver. While standard networks prioritize low-frequency information, model loss—due to its reliance on derivatives—shifts focus toward higher frequencies, potentially destabilizing the learned low-frequency components.
Introduction
Using neural networks to solve Partial Differential Equations (PDEs) is a cornerstone of modern scientific machine learning. Typically, these models are trained using two types of information: Data Loss (direct observations of the solution) and Model Loss (the governing physics equations, such as derivatives and boundary conditions). A common strategy is to "pre-train" a network on data and then refine it using the physics equations. However, researchers have identified that this transition is often not smooth. Instead, the network frequently "jumps" away from the correct solution immediately after the switch, a phenomenon termed the "loss-jump."
Methodology
The authors combined empirical experiments with rigorous mathematical analysis to understand this behavior:
- Empirical Testing: Neural networks were trained on several classic PDEs, including Poisson, Burgers, Heat, Diffusion, and Wave equations. The "loss jump" was monitored by training for a set number of epochs on data loss before switching entirely to model loss.
- Spectral Analysis: Fourier transforms were employed to track how error at different frequencies (low vs. high) evolved during the transition.
- Theoretical Modeling: The authors derived training dynamics for a two-layer, infinite-width neural network in the Neural Tangent Kernel (NTK) regime to analyze convergence rates of different frequency components under varying loss functions.
Key Insights: Conflicting Spectral Biases
The core finding is that data loss and model loss have fundamentally different "frequency preferences."
- The Loss-Jump Phenomenon: The jump is a stable, repeatable phenomenon where the model abandons a high-accuracy state because the new loss function perceives the error landscape differently. It is not merely an artifact of high learning rates or optimizer shocks.
- Frequency Principle vs. Derivative Amplification: Standard neural networks follow a "Frequency Principle," fitting low frequencies first. However, model loss involves derivatives, which mathematically amplify high-frequency signals. This shifts the network's attention toward medium and high frequencies, leaving low-frequency errors weakly constrained.
- The "Sweet Spot": Theoretical analysis shows that for model loss, the convergence rate peaks at medium frequencies. Consequently, the network may ignore a large, slow-moving error (low frequency) to fix a tiny, fast-moving one (high frequency).
Results and Discussion
In experiments with the Poisson equation, switching to model loss caused the data loss (the true error) to spike instantly, regardless of the learning rate. For the Heat and Diffusion equations, error heatmaps showed that the "minimum" found by data loss is not treated as a minimum by model loss, forcing the network to migrate to a different, often less accurate, region of the parameter space.
As shown in the solution profiles, the model's prediction (dashed red) begins to deviate significantly from the true solution (solid blue) as training progresses under model loss, despite the model loss itself decreasing. This confirms that the network "forgets" the low-frequency accuracy gained during pre-training.
Conclusion
The discovery of the loss-jump phenomenon suggests that simple pre-training strategies for PINNs may be insufficient. Future work must focus on developing adaptive training strategies, such as frequency-dependent weighting schemes, to ensure that neural networks maintain low-frequency accuracy while incorporating high-frequency physics constraints. While this study focused on 1D and 2D benchmarks, the spectral bias identified likely plays a critical role in more complex, high-dimensional systems.
Summary updated