Can Transfer Neuroevolution Tractably Solve Your Differential Equations?

Jian Cheng Wong, Abhishek Gupta, Yew-Soon Ong

TL;DR

This paper proposes a novel "transfer neuroevolution" method to solve differential equations more effectively. By combining neuroevolution's ability to avoid local optima with a transfer learning mechanism that reuses knowledge from similar problems, the approach achieves better accuracy and faster convergence than traditional gradient descent methods like SGD.

What Problem Does This Paper Address?

Solving differential equations is fundamental to scientific modeling. A recent approach, Physics-Informed Neural Networks (PINNs), uses deep neural networks to approximate solutions. The challenge lies in optimizing these PINNs. Traditional methods like Stochastic Gradient Descent (SGD) can get stuck in local minima, leading to inaccurate solutions, which is problematic when the goal is a globally optimal, physically accurate solution. Neuroevolution, which explores multiple solutions in parallel, can circumvent local optima but is often slow. This paper addresses the need for a more efficient and robust optimization method for PINNs that can achieve globally optimal solutions.

Methods

The core of the paper introduces Transfer Neuroevolution (tNES). This method builds upon neuroevolution, specifically using Natural Evolution Strategies (NES).

  • Neuroevolution (NES): Instead of gradient descent, NES uses a population of candidate solutions and evolves a probabilistic model (a search distribution) to find optimal neural network weights. This parallel exploration helps avoid local optima.
  • Transfer Learning: To address the slowness of neuroevolution, tNES incorporates a mixture model-based adaptive transfer mechanism. This allows the algorithm to leverage "experiential priors" (learned parameters from solving similar problems).
  • Adaptive Transfer: The mixture model dynamically adjusts the influence of source problems, prioritizing those that are most beneficial and mitigating "negative transfer" (where knowledge from a source problem harms performance on the target problem).
  • Pseudo-offspring Projection: To further enhance stability, pseudo-offspring sampled from source distributions are projected closer to the target search distribution's natural coordinates, preventing overly drastic updates.

Datasets

The paper does not introduce new datasets. Instead, it uses a variety of well-established differential equations as benchmarks for evaluation. These include:

  • 1D steady-state convection-diffusion equation
  • 2D projectile motion equations
  • Model equations of traveling waves (Linearized Burgers', Nonlinear Burgers', and Korteweg-de Vries (KdV) equations)

These problems are varied to test the robustness of the proposed method across different types of differential equations and scenarios (e.g., changing parameters like velocity, gravity, or diffusion coefficients).

Key Insights and Ideas

  • Global Optimization is Key for PDEs: Unlike standard deep learning where out-of-sample generalization is paramount, solving differential equations requires finding the globally optimal parameters that satisfy the physics. SGD's tendency to settle for good-enough local optima is a significant drawback.
  • Neuroevolution's Parallel Exploration is Powerful: The population-based approach of neuroevolution naturally lends itself to exploring diverse solution landscapes, making it more adept at finding global optima than single-path gradient descent.
  • Transfer Learning is Crucial for Tractability: While neuroevolution is effective, its computational cost is a barrier. Transfer learning, by reusing knowledge from related problems, significantly speeds up convergence and improves efficiency, making neuroevolution "tractable."
  • Adaptive Transfer Prevents Negative Transfer: The proposed adaptive mechanism is critical. It allows the algorithm to intelligently select and weigh knowledge from multiple source problems, actively discarding irrelevant or harmful information.
  • Physics-Informed Neural Networks are a Global Optimization Problem: The paper frames PINN training not as a typical machine learning task but as a global optimization challenge, where the loss function directly encodes the physical laws.

Key Results

The paper presents empirical evidence demonstrating the effectiveness of tNES across several differential equations:

  • Improved Accuracy: tNES consistently achieved better optimized loss (lower residuals) compared to both SGD (ADAM) and baseline neuroevolution (xNES). In some cases, the difference was orders of magnitude.
  • Faster Convergence: tNES significantly sped up convergence, particularly in challenging scenarios where SGD struggled or got trapped. This was especially noticeable under limited computational budgets (e.g., 50k evaluations).
  • Robustness: The method showed robustness across various problem parameters and types, including convection-diffusion, projectile motion, and traveling wave equations.
  • Overcoming SGD Limitations: In problems with high velocity or steep gradients, SGD often produced unphysical or meaningless solutions, whereas tNES maintained accuracy.

Limitations and Future Work

  • Handling Larger Networks: The authors acknowledge that neuroevolution might face challenges with significantly larger neural networks, suggesting future work on improving its scalability for more complex differential equations.
  • Automated Architecture Search: Currently, the neural network architecture is fixed. Future research could explore integrating architecture search with neuroevolution for end-to-end optimization.
  • Multi-Objective and Multi-Task Neuroevolution: The paper suggests exploring multi-objective and multi-task learning paradigms within neuroevolution for PINNs.
  • Domain Decomposition: Combining tNES with domain decomposition strategies, where different neural networks handle different sub-domains, is proposed as a promising avenue.
  • Parameter Sensitivity: While the adaptive transfer mechanism is robust, the authors note that tNES, like its baseline xNES, can still be sensitive to certain tunable parameters (e.g., population size, learning rates), and optimal settings may vary across problems.

Summary updated

Abstract

This paper introduces neuroevolution for solving differential equations. The solution is obtained through optimizing a deep neural network whose loss function is defined by the residual terms from the differential equations. Recent studies have focused on learning such physics-informed neural networks through stochastic gradient descent (SGD) variants, yet they face the difficulty of obtaining an accurate solution due to optimization challenges. In the context of solving differential equations, we are faced with the problem of finding globally optimum parameters of the network, instead of being concerned with out-of-sample generalization. SGD, which searches along a single gradient direction, is prone to become trapped in local optima, so it may not be the best approach here. In contrast, neuroevolution carries out a parallel exploration of diverse solutions with the goal of circumventing local optima. It could potentially find more accurate solutions with better optimized neural networks. However, neuroevolution can be slow, raising tractability issues in practice. With that in mind, a novel and computationally efficient transfer neuroevolution algorithm is proposed in this paper. Our method is capable of exploiting relevant experiential priors when solving a new problem, with adaptation to protect against the risk of negative transfer. The algorithm is applied on a variety of differential equations to empirically demonstrate that transfer neuroevolution can indeed achieve better accuracy and faster convergence than SGD. The experimental outcomes thus establish transfer neuroevolution as a noteworthy approach for solving differential equations, one that has never been studied in the past. Our work expands the resource of available algorithms for optimizing physics-informed neural networks.
BibTeX
@inproceedings{wong2021can,
  author = {Jian Cheng Wong and Abhishek Gupta and Yew-Soon Ong},
  title = {{Can Transfer Neuroevolution Tractably Solve Your Differential Equations?}},
  booktitle = {IEEE Comput. Intell. Mag},
  year = {2021},
  url = {https://openreview.net/forum?id=ltWcPV5Bvt},
}
Figure