NSGA-PINN: A Multi-Objective Optimization Method for Physics-Informed Neural Network Training

Guang Lin, Binghang Lu, Christian Moya, Guang Lin 0001Algorithms 2023 · DOI

TL;DR

This paper introduces NSGA-PINN, a novel framework that leverages the multi-objective optimization capabilities of the NSGA-II algorithm to improve the training of Physics-Informed Neural Networks (PINNs). By treating each component of the PINN loss function as a distinct objective, NSGA-PINN effectively helps PINNs escape local minima and more precisely satisfy initial and boundary conditions, especially in challenging inverse problems with noisy data.

What Problem Does This Paper Address?

Physics-Informed Neural Networks (PINNs) have emerged as a powerful tool for solving differential equations, integrating physical laws directly into the neural network's training process. However, a significant challenge in training PINNs lies in optimizing their complex, often non-convex loss functions. Traditional methods like Stochastic Gradient Descent (SGD) and its variants (e.g., ADAM) can easily get trapped in suboptimal local minima, particularly when dealing with inverse problems or noisy experimental data. Furthermore, these gradient-based methods treat initial and boundary conditions as "soft constraints," meaning they are not always perfectly satisfied, which can be problematic for applications requiring exact adherence to these conditions. This paper aims to address these limitations by proposing a more robust optimization strategy.

Methods

The core innovation of this paper is the NSGA-PINN framework. Instead of minimizing a single, weighted sum of all loss components (residual, initial, boundary, and data loss), NSGA-PINN treats each of these loss components as a separate objective to be minimized simultaneously.

The framework integrates the Non-dominated Sorting Genetic Algorithm II (NSGA-II) with traditional optimization methods. Here's how it works:

  • Multi-Objective Formulation: The PINN's total loss is decomposed into multiple objectives (e.g., $L_{res}$, $L_{ic}$, $L_{bc}$, $L_{data}$).
  • NSGA-II for Global Search: NSGA-II is employed to explore the search space of potential PINN solutions. It uses non-dominated sorting to rank solutions based on their Pareto optimality and crowding distance to maintain diversity within the population. This helps the algorithm avoid getting stuck in local minima.
  • Hybrid Approach: After NSGA-II identifies promising candidate solutions, a traditional optimizer like ADAM is used to further refine these solutions and reduce their loss values. This hybrid approach aims to combine the global search capabilities of NSGA-II with the efficient local optimization of ADAM.
  • Crowded Binary Tournament Selection: This selection mechanism is used within NSGA-II to choose the best individuals for the next generation, prioritizing solutions with lower front values (better Pareto dominance) and higher density values (better diversity).

Datasets

The paper does not explicitly mention or introduce new datasets. Instead, it uses synthetic data generated from the solutions of the differential equations being solved. The experimental setup involves:

  • Inverse Pendulum Problem: Data points for the pendulum's angle and velocity are generated based on known initial conditions and a true parameter value.
  • Burgers Equation: Initial and boundary conditions are defined, and the spatial domain is discretized to generate data points for training and evaluation.
  • Noisy Data: For inverse problems, Gaussian noise is intentionally added to the generated data to test the robustness of the proposed method.

Key Insights and Ideas

  • Decomposing Loss is Key: Treating each loss component as a distinct objective allows the optimization algorithm to balance competing requirements more effectively than a simple weighted sum. This is particularly crucial when initial/boundary conditions need to be satisfied precisely.
  • Escaping Local Minima: The multi-objective nature and population-based search of NSGA-II are highly effective at preventing the optimizer from getting trapped in local optima, a common pitfall for gradient-based methods in complex PINN landscapes.
  • Robustness to Noise: NSGA-PINN demonstrates superior performance compared to ADAM when dealing with noisy input data, achieving lower total loss and better adherence to physical constraints.
  • Ensemble Benefits: While computationally more expensive, the ensemble approach inherent in NSGA-II provides a more reliable and robust solution, especially when quantifying uncertainty through confidence intervals.

Key Results

The NSGA-PINN framework consistently outperformed the traditional ADAM optimizer across the tested problems:

  • Inverse Pendulum Problem (No Noise): NSGA-PINN achieved a total loss of $6.55 \times 10^{-5}$, significantly lower than ADAM's $1.935 \times 10^{-4}$, indicating better satisfaction of initial conditions.
  • Inverse Pendulum Problem (Noisy Data): NSGA-PINN achieved a total loss of $0.0133$, outperforming ADAM's $0.017$. The method also produced results closer to the true solution in this noisy scenario.
  • Burgers Equation: NSGA-PINN achieved a total loss of $3.746 \times 10^{-5}$, vastly superior to ADAM's $0.0003$.
  • Burgers Equation (Noisy Data): NSGA-PINN achieved a total loss of $0.0061$, compared to ADAM's $0.0526$. The NSGA-PINN predictions closely matched the true solution.
  • Survival Rate: The survival rate of the NSGA-PINN algorithm increased from approximately 50% in early generations to 73% by the end, indicating progressive learning and effective utilization of learned results.

Limitations and Future Work

  • Computational Cost: The primary limitation acknowledged is the increased computational cost due to the ensemble nature of NSGA-II, requiring the training of multiple neural networks.
  • Hyperparameter Tuning: The parent population size, maximum generation number, and epochs for the ADAM optimizer are manually initialized, suggesting potential for further optimization in hyperparameter selection.
  • Future Work: The authors plan to explore ways to mitigate the computational cost, potentially through parallelization. They also aim to derive effective methods for finding the optimal trade-off between the NSGA and ADAM components. Further research will focus on balancing the parent population size, maximum generation number, and ADAM epochs to optimize performance and prevent overfitting.

Summary updated

BibTeX
@article{lin2023nsga,
  author = {Guang Lin and Binghang Lu and Christian Moya and Guang Lin},
  title = {{NSGA-PINN: A Multi-Objective Optimization Method for Physics-Informed Neural Network Training}},
  journal = {Algorithms},
  year = {2023},
  doi = {10.3390/a16040194},
}
Figure