JAX-Accelerated Neuroevolution of Physics-informed Neural Networks: Benchmarks and Experimental Results

Nicholas Wei Yong Sung, Jian Cheng Wong, Pao-Hsiung Chiu, Abhishek Gupta, Chinchun Ooi, Yew-Soon Ong

TL;DR

This paper argues that traditional gradient descent methods struggle with the complex loss landscapes of Physics-Informed Neural Networks (PINNs), leading to suboptimal solutions. They propose neuroevolutionary algorithms as a more suitable alternative and introduce a new benchmark suite to demonstrate that these methods can achieve better physics compliance and faster convergence than gradient descent.

What Problem Does This Paper Address?

Physics-Informed Neural Networks (PINNs) are a powerful tool for solving scientific and engineering problems by embedding physical laws directly into the neural network's loss function. However, training PINNs is notoriously difficult. Unlike standard deep learning tasks where finding a good solution is often sufficient, PINNs require finding a globally optimal solution that precisely adheres to physical principles. The authors highlight that the unique loss formulations of PINNs create highly complex and "rugged" loss landscapes, making them prone to getting stuck in "spurious local minima" – solutions that satisfy some physics but not others, leading to inaccurate predictions. Traditional gradient descent methods, which are good at finding local optima, struggle in such environments.

Methods

The core of this paper is a comparative study of optimization algorithms for training PINNs. The authors contrast:

  • Stochastic Gradient Descent (SGD): The standard workhorse for training deep learning models, including PINNs.
  • Neuroevolutionary Algorithms: Specifically, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) and exponential Natural Evolution Strategies (xNES), augmented with Nesterov Accelerated Gradient (xNES+NAG). These are population-based, gradient-free methods known for their ability to explore complex search spaces and escape local minima.

They implement these algorithms using the JAX framework, which allows for flexible definition and efficient training of neural networks, especially on hardware accelerators. A key contribution is the development of a benchmark suite of five diverse PINN problems, complete with open-source code and ground truth solutions, to facilitate systematic evaluation.

Datasets

The paper does not use traditional "datasets" in the sense of labeled examples. Instead, it introduces a novel benchmark suite consisting of five physics problems:

  • Steady state convection-diffusion equation
  • Projectile motion
  • Korteweg-De Vries (KdV) equation
  • Linearized Burgers' equation
  • Non-linear Burgers' equation

These problems cover various physical phenomena and are used to evaluate the performance of different optimizers. Ground truth solutions for these problems are provided, either analytically or through numerical simulations.

Key Insights and Ideas

  • PINN Loss Landscapes are Inherently Rugged: The paper provides compelling visual evidence (Figure 1 and 2) showing that PINN loss landscapes are significantly more complex and prone to local minima than those of standard deep neural networks trained on data. This complexity stems directly from embedding PDEs into the loss function.
  • Initialization is a Major Hurdle for PINNs: The authors demonstrate that PINNs can be trapped in "deceptive local minima" right at initialization, where the PDE residual is zero but boundary/initial conditions are violated. This makes it very difficult for gradient descent to even start finding a valid solution.
  • Neuroevolution Offers Superior Global Exploration: The population-based nature of neuroevolutionary algorithms allows them to explore the loss landscape more broadly, making them better equipped to avoid or escape these spurious local minima compared to gradient descent.
  • A Standardized Benchmark is Crucial: The introduction of a comprehensive, open-source benchmark suite is a significant contribution, enabling reproducible and standardized comparisons of optimization methods for PINNs.

Key Results

  • Faster Convergence: Neuroevolutionary algorithms (CMA-ES and xNES+NAG) generally converged to lower training losses much faster than SGD across the five benchmark problems (Figure 4).
  • Better Physics Compliance (Lower MSE): Crucially, neuroevolutionary algorithms consistently achieved the lowest Prediction Mean Squared Error (MSE) against the ground truth solutions for four out of five benchmark problems, indicating better adherence to physical laws (Table 4). CMA-ES, in particular, showed strong performance.
  • Visual Evidence of Superiority: Plots of the solutions generated by CMA-ES closely matched the ground truth, while SGD solutions were often less accurate, especially for complex problems like KdV and non-linear Burgers (Figures 5 and 6).

Limitations and Future Work

  • Generalizability: While the results are promising, the authors acknowledge that these findings do not conclusively guarantee that EAs will surpass SGD for all PINN problems. Further research is needed to explore a wider range of problems and PINN architectures.
  • Hybrid Approaches: The paper notes an intersection in convergence curves between optimizers, suggesting that hybrid approaches combining global search (neuroevolution) with local gradient information could be a promising avenue for future research.
  • Algorithm Development: The benchmark suite is intended to motivate the development of novel neuroevolutionary algorithms specifically tailored for PINN optimization.

Summary updated

Abstract

The potential of learned models for fundamental scientific research and discovery is drawing increasing attention worldwide. Physics-informed neural networks (PINNs), where the loss function directly embeds governing equations of scientific phenomena, is one of the key techniques at the forefront of recent advances. PINNs are typically trained using stochastic gradient descent methods, akin to their deep learning counterparts. However, analysis in this paper shows that PINNs' unique loss formulations lead to a high degree of complexity and ruggedness that may not be conducive for gradient descent. Unlike in standard deep learning, PINN training requires globally optimum parameter values that satisfy physical laws as closely as possible. Spurious local optimum, indicative of erroneous physics, must be avoided. Hence, neuroevolution algorithms, with their superior global search capacity, may be a better choice for PINNs relative to gradient descent methods. Here, we propose a set of five benchmark problems, with open-source codes, spanning diverse physical phenomena for novel neuroevolution algorithm development. Using this, we compare two neuroevolution algorithms against the commonly used stochastic gradient descent, and our baseline results support the claim that neuroevolution can surpass gradient descent, ensuring better physics compliance in the predicted outputs. %Furthermore, implementing neuroevolution with JAX leads to orders of magnitude speedup relative to standard implementations.
BibTeX
@inproceedings{sung2023jax,
  author = {Nicholas Wei Yong Sung and Jian Cheng Wong and Pao-Hsiung Chiu and Abhishek Gupta and Chinchun Ooi and Yew-Soon Ong},
  title = {{JAX-Accelerated Neuroevolution of Physics-informed Neural Networks: Benchmarks and Experimental Results}},
  booktitle = {GECCO Companion},
  year = {2023},
  url = {https://openreview.net/forum?id=sQ0ngL5ldQ},
}
Figure