Loss Jump During Loss Switch in Solving PDEs with Neural Networks

Zhiwei Wang, Lulu Zhang, Zhongwang Zhang, Zhi-Qin John XuarXiv 2024 · DOI · arXiv

Abstract

The "loss-jump" phenomenon in physics-informed neural networks (PINNs) occurs when transitioning training from data-based supervision to physics-based constraints (PDE residuals). This transition often leads to a significant, sudden decrease in model accuracy. This study investigates the underlying causes, identifying conflicting spectral biases between data loss and model loss as the primary driver. While standard networks prioritize low-frequency information, model loss—due to its reliance on derivatives—shifts focus toward higher frequencies, potentially destabilizing the learned low-frequency components.

Introduction

Using neural networks to solve Partial Differential Equations (PDEs) is a cornerstone of modern scientific machine learning. Typically, these models are trained using two types of information: Data Loss (direct observations of the solution) and Model Loss (the governing physics equations, such as derivatives and boundary conditions). A common strategy is to "pre-train" a network on data and then refine it using the physics equations. However, researchers have identified that this transition is often not smooth. Instead, the network frequently "jumps" away from the correct solution immediately after the switch, a phenomenon termed the "loss-jump."

Methodology

The authors combined empirical experiments with rigorous mathematical analysis to understand this behavior:

  • Empirical Testing: Neural networks were trained on several classic PDEs, including Poisson, Burgers, Heat, Diffusion, and Wave equations. The "loss jump" was monitored by training for a set number of epochs on data loss before switching entirely to model loss.
  • Spectral Analysis: Fourier transforms were employed to track how error at different frequencies (low vs. high) evolved during the transition.
  • Theoretical Modeling: The authors derived training dynamics for a two-layer, infinite-width neural network in the Neural Tangent Kernel (NTK) regime to analyze convergence rates of different frequency components under varying loss functions.

Key Insights: Conflicting Spectral Biases

The core finding is that data loss and model loss have fundamentally different "frequency preferences."

  1. The Loss-Jump Phenomenon: The jump is a stable, repeatable phenomenon where the model abandons a high-accuracy state because the new loss function perceives the error landscape differently. It is not merely an artifact of high learning rates or optimizer shocks.
Figure 1: The loss-jump phenomenon in a Poisson problem using tanh activation. Colored lines show the trajectory of data loss after switching from data-based pre-training to model loss at different epochs (stars).
  1. Frequency Principle vs. Derivative Amplification: Standard neural networks follow a "Frequency Principle," fitting low frequencies first. However, model loss involves derivatives, which mathematically amplify high-frequency signals. This shifts the network's attention toward medium and high frequencies, leaving low-frequency errors weakly constrained.
  2. The "Sweet Spot": Theoretical analysis shows that for model loss, the convergence rate peaks at medium frequencies. Consequently, the network may ignore a large, slow-moving error (low frequency) to fix a tiny, fast-moving one (high frequency).

Results and Discussion

In experiments with the Poisson equation, switching to model loss caused the data loss (the true error) to spike instantly, regardless of the learning rate. For the Heat and Diffusion equations, error heatmaps showed that the "minimum" found by data loss is not treated as a minimum by model loss, forcing the network to migrate to a different, often less accurate, region of the parameter space.

Figure 5: Evolution of loss and solution profiles after switching from data to model loss. The blue line shows data loss increasing as the model loss (red) is minimized, leading to solution divergence.

As shown in the solution profiles, the model's prediction (dashed red) begins to deviate significantly from the true solution (solid blue) as training progresses under model loss, despite the model loss itself decreasing. This confirms that the network "forgets" the low-frequency accuracy gained during pre-training.

Conclusion

The discovery of the loss-jump phenomenon suggests that simple pre-training strategies for PINNs may be insufficient. Future work must focus on developing adaptive training strategies, such as frequency-dependent weighting schemes, to ensure that neural networks maintain low-frequency accuracy while incorporating high-frequency physics constraints. While this study focused on 1D and 2D benchmarks, the spectral bias identified likely plays a critical role in more complex, high-dimensional systems.

Summary updated

Abstract

Using neural networks to solve partial differential equations (PDEs) is gaining popularity as an alternative approach in the scientific computing community. Neural networks can integrate different types of information into the loss function. These include observation data, governing equations, and variational forms, etc. These loss functions can be broadly categorized into two types: observation data loss directly constrains and measures the model output, while other loss functions indirectly model the performance of the network, which can be classified as model loss. However, this alternative approach lacks a thorough understanding of its underlying mechanisms, including theoretical foundations and rigorous characterization of various phenomena. This work focuses on investigating how different loss functions impact the training of neural networks for solving PDEs. We discover a stable loss-jump phenomenon: when switching the loss function from the data loss to the model loss, which includes different orders of derivative information, the neural network solution significantly deviates from the exact solution immediately. Further experiments reveal that this phenomenon arises from the different frequency preferences of neural networks under different loss functions. We theoretically analyze the frequency preference of neural networks under model loss. This loss-jump phenomenon provides a valuable perspective for examining the underlying mechanisms of neural networks in solving PDEs.
BibTeX
@misc{wang2024loss,
  author = {Zhiwei Wang and Lulu Zhang and Zhongwang Zhang and Zhi-Qin John Xu},
  title = {{Loss Jump During Loss Switch in Solving PDEs with Neural Networks}},
  year = {2024},
  eprint = {2405.03095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.03095},
}
Figure