Generalizable Neural Physics Solvers by Baldwinian Evolution

Jian Cheng Wong, Chin Chun Ooi, Abhishek Gupta, Pao-Hsiung Chiu, Joshua Shao Zheng Low, My Ha Dao, Yew-Soon OngarXiv 2023 · DOI · arXiv · OpenReview

TL;DR

This paper introduces "Baldwinian-PINNs," a novel approach to make Physics-Informed Neural Networks (PINNs) generalize across a wide range of physics problems without costly retraining. By combining evolutionary optimization with a specialized "lifetime learning" phase, these PINNs are pre-wired to learn physics efficiently, achieving significant speedups and accuracy improvements over existing meta-learning methods.

What Problem Does This Paper Address?

Physics-Informed Neural Networks (PINNs) are powerful tools for scientific machine learning, integrating physical laws directly into the learning process. However, a major limitation is their lack of generalizability: a PINN trained for one specific physics problem (e.g., a particular PDE with specific parameters) often requires extensive, computationally expensive retraining to solve a new, even slightly different, problem. This makes them impractical for applications requiring rapid predictions across many scenarios. Existing meta-learning approaches for PINNs often still involve significant retraining or rely on task parameterization, which can be restrictive. The authors aim to overcome this by developing PINNs that are inherently equipped to learn diverse physics tasks efficiently, inspired by the Baldwin effect in evolutionary biology.

Methods

The core innovation is the "Baldwinian evolution" framework applied to PINNs. This framework is implemented as a two-stage stochastic programming problem:

  1. Evolutionary Optimization (Outer Loop): A population of PINN models (specifically, their initial hidden layer weights and learning hyperparameters) is evolved over generations. The "fitness" of each model is determined by its performance on a distribution of physics tasks after a specialized "lifetime learning" phase. This evolutionary process biases the initial network structure towards efficient physics learning.
  2. Lifetime Learning (Inner Loop): For each individual PINN in the evolutionary population, its final layer weights are rapidly trained to specialize on a sampled physics task. This is achieved using a physics-informed least-squares formulation with Tikhonov regularization. Crucially, this "lifetime learning" is very fast (order of milliseconds) and only affects the final layer, leaving the evolved hidden layer structure intact.

This approach mimics the Baldwin effect, where learned traits (lifetime learning) are eventually reinforced in the genetic makeup (evolved initial weights) of the population over generations, leading to inherent predispositions for learning.

Datasets

The paper evaluates Baldwinian-PINNs on a variety of benchmark physics problems, including:

  • Convection-diffusion equation: Used for demonstrating generalization across different difficulty levels.
  • Linear ODE/PDEs: Including 1D and 2D Poisson's equations, Helmholtz equation, and a family of linear PDEs (convection, diffusion, dispersion).
  • Nonlinear ODE/PDEs: Kinematics (projectile motion), Burgers' equation, nonlinear heat equation, Allen-Cahn equation, and diffusion-reaction equations.
  • Navier-Stokes equations: For fluid flow in a lid-driven cavity, representing a challenging nonlinear problem.

The authors generated diverse training and test task distributions for each problem, varying parameters, initial conditions (ICs), and boundary conditions (BCs) to rigorously test generalization capabilities.

Key Insights and Ideas

  • Baldwin Effect for PINNs: The paper successfully translates the biological Baldwin effect into a computational framework for PINNs, showing that evolving initial network biases can lead to superior generalization compared to traditional meta-learning.
  • Fast Lifetime Learning: The use of Tikhonov regularization for the final layer specialization allows for extremely rapid "lifetime learning" (milliseconds), enabling quick adaptation to new tasks.
  • Parameterization-Agnostic Generalization: Baldwinian-PINNs do not require explicit input parameterization for new tasks, making them more flexible than many existing meta-learning methods.
  • Pre-wired for Physics: The evolutionary process effectively "pre-wires" the PINNs with strong learning biases, making them inherently better at learning physics from the outset.
  • Order-of-Magnitude Improvements: The approach demonstrates significant gains in both prediction accuracy and computational speed compared to state-of-the-art methods.

Key Results

  • Accuracy Improvement: Achieved up to an order of magnitude improvement in prediction accuracy compared to recent meta-learned PINNs. For instance, a 70x accuracy improvement was observed for the diffusion-reaction equation.
  • Speedup: Demonstrated a substantial reduction in computational cost. For the diffusion-reaction equation, it took 700x less computational time. Predictions for new tasks are achieved in milliseconds.
  • Generalization: Successfully generalized across diverse problem instances, including variations in PDE parameters, ICs, BCs, and even extended time domains.
  • Robustness: Showed resilience to noisy or scarce training data.
  • Navier-Stokes: Achieved accurate solutions for the challenging Navier-Stokes equations in under a second, even for high Reynolds numbers.

Limitations and Future Work

  • Computational Cost of Evolution: While lifetime learning is fast, the evolutionary optimization process itself can be computationally intensive, though it is highly parallelizable.
  • Hidden Layer Adaptation: The current framework keeps the hidden layers fixed after evolution. The authors suggest that allowing light hidden-layer adaptation (e.g., a few gradient-descent steps on a low-rank adapter) could offer more flexibility for out-of-distribution tasks, albeit at the cost of some speed.
  • Neural Architecture Search: The paper focuses on optimizing the distribution parameters of weights. Future work could integrate more advanced neural architecture search techniques to optimize the network's graph structure.
  • Broader Applicability: The authors express interest in testing the limits of Baldwinian-PINNs for learning across a wider range of physics phenomena and differential operators.

Summary updated

Abstract

Physics-informed neural networks (PINNs) are at the forefront of scientific machine learning, making possible the creation of machine intelligence that is cognizant of physical laws and able to accurately simulate them. However, today's PINNs are often trained for a single physics task and require computationally expensive re-training for each new task, even for tasks from similar physics domains. To address this limitation, this paper proposes a pioneering approach to advance the generalizability of PINNs through the framework of Baldwinian evolution. Drawing inspiration from the neurodevelopment of precocial species that have evolved to learn, predict and react quickly to their environment, we envision PINNs that are pre-wired with connection strengths inducing strong biases towards efficient learning of physics. A novel two-stage stochastic programming formulation coupling evolutionary selection pressure (based on proficiency over a distribution of physics tasks) with lifetime learning (to specialize on a sampled subset of those tasks) is proposed to instantiate the Baldwin effect. The evolved Baldwinian-PINNs demonstrate fast and physics-compliant prediction capabilities across a range of empirically challenging problem instances with more than an order of magnitude improvement in prediction accuracy at a fraction of the computation cost compared to state-of-the-art gradient-based meta-learning methods. For example, when solving the diffusion-reaction equation, a 70x improvement in accuracy was obtained while taking 700x less computational time. This paper thus marks a leap forward in the meta-learning of PINNs as generalizable physics solvers. Sample codes are available at https://github.com/chiuph/Baldwinian-PINN.
BibTeX
@misc{wong2023generalizable,
  author = {Jian Cheng Wong and Chin Chun Ooi and Abhishek Gupta and Pao-Hsiung Chiu and Joshua Shao Zheng Low and My Ha Dao and Yew-Soon Ong},
  title = {{Generalizable Neural Physics Solvers by Baldwinian Evolution}},
  year = {2023},
  eprint = {2312.03243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.03243},
}
Figure