Bayesian Learning to Discover Mathematical Operations in Governing Equations of Dynamic Systems

Hongpeng Zhou, Wei Pan

Abstract

Discovering governing equations from data is critical for diverse scientific disciplines as they can provide insights into the underlying phenomenon of dynamic systems. This work presents a new representation for governing equations by designing the Mathematical Operation Network (MathONet) with a deep neural network-like hierarchical structure. Specifically, the MathONet is stacked by several layers of unary operations (e.g., sin, cos, log) and binary operations (e.g., +,-), respectively. An initialized MathONet is typically regarded as a super-graph with a redundant structure, a sub-graph of which can yield the governing equation. We develop a sparse group Bayesian learning algorithm to extract the sub-graph by employing structurally constructed priors over the redundant mathematical operations. By demonstrating the chaotic Lorenz system, Lotka-Volterra system, and Kolmogorov-Petrovsky-Piskunov system, the proposed method can discover the ordinary differential equations (ODEs) and partial differential equations (PDEs) from the observations given limited mathematical operations, without any prior knowledge on possible expressions of the ODEs and PDEs.

Introduction

In science and engineering, identifying the "governing equations"—the mathematical laws describing how a system evolves—is essential for prediction and control. Traditionally, these are derived from first principles, but many complex systems lack clear underlying laws.

Existing data-driven methods face significant limitations. Symbolic regression is computationally expensive and prone to overfitting, while sparse regression techniques like SINDy require a predefined "dictionary" of candidate functions. If the true term is missing from the dictionary, the model fails. Conversely, standard neural networks act as black boxes, offering high predictive power but no interpretability. The researchers behind MathONet aim to bridge this gap by creating a system that learns the structure of mathematical operations directly from data.

Methodology: The MathONet Framework

The authors introduce a novel architecture called the Mathematical Operation Network (MathONet). Instead of standard neurons, MathONet utilizes layers of unary operations (e.g., $\sin, \cos, \log, \exp$) and binary operations (e.g., $+, -, \times$).

1. Network Architecture

MathONet is structured similarly to a DenseNet, where the output of every layer is "pasted" into all subsequent layers. This allows the network to build increasingly complex compositions (such as $\sin(x^2 + y)$) from basic building blocks. The network begins as an over-parameterized "super-graph"—a dense web of all possible mathematical combinations.

The hierarchical architecture of MathONet, featuring Polynomial and Operation networks within the layers.

2. Sparse Group Bayesian Learning

To identify the true governing equation, the redundant connections in the super-graph must be pruned. The authors developed a Bayesian approach using Laplace approximation to assign "sparsity priors" to weights. As the model trains, it calculates the "evidence" for each operation. If an operation does not significantly help explain the data, its weight is driven to zero, and it is pruned from the graph. This frames scientific discovery as a network compression problem.

Key Results and Insights

The model was evaluated on classic benchmarks, including the chaotic Lorenz system, the Lotka-Volterra predator-prey model, and the Fisher-KPP partial differential equation (PDE).

Discovery as Compression

In the Lorenz system experiment, the model started with 1,551 possible terms. After six cycles of Bayesian pruning, it successfully reduced the model to the exact two terms required for the $z$-axis equation, significantly outperforming standard techniques like Sparse Group Lasso.

The pruning process over cycle indices, showing the reduction from 1,551 terms to the 2 exact terms of the governing equation.

Robustness and PDE Discovery

The method remained accurate even with significant Gaussian noise added to the data, identifying correct terms within 0.5% of their true values. Furthermore, by combining MathONet with CNN-style spatial stencils, the model successfully discovered the Fisher-KPP equation, identifying both the diffusion coefficient and the growth polynomial accurately.

Conclusion

MathONet represents a shift from picking functions from a list to composing them from fundamental operations. While the current model requires users to define initial network depth and faces computational costs related to Hessian matrix calculations, it provides a principled, Bayesian way to discover interpretable laws of nature from raw observations.

Summary updated

Abstract

Discovering governing equations from data is critical for diverse scientific disciplines as they can provide insights into the underlying phenomenon of dynamic systems. This work presents a new representation for governing equations by designing the Mathematical Operation Network (MathONet) with a deep neural network-like hierarchical structure. Specifically, the MathONet is stacked by several layers of unary operations (e.g., sin, cos, log) and binary operations (e.g., +,-), respectively. An initialized MathONet is typically regarded as a super-graph with a redundant structure, a sub-graph of which can yield the governing equation. We develop a sparse group Bayesian learning algorithm to extract the sub-graph by employing structurally constructed priors over the redundant mathematical operations. By demonstrating the chaotic Lorenz system, Lotka-Volterra system, and Kolmogorov-Petrovsky-Piskunov system, the proposed method can discover the ordinary differential equations (ODEs) and partial differential equations (PDEs) from the observations given limited mathematical operations, without any prior knowledge on possible expressions of the ODEs and PDEs.
BibTeX
@misc{zhou2022bayesian,
  author = {Hongpeng Zhou and Wei Pan},
  title = {{Bayesian Learning to Discover Mathematical Operations in Governing Equations of Dynamic Systems}},
  year = {2022},
  eprint = {2206.00669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.00669},
}
Figure