TL;DR
This paper introduces a machine learning framework for discovering coarse-grained equations from data, particularly for probability density functions (PDFs) of complex systems. By leveraging sparse regression and physics-informed dictionaries, the method can learn governing equations, including closure approximations, from simulated data, offering a powerful tool for multiscale modeling and stochastic simulations.
What Problem Does This Paper Address?
Discovering governing equations from data is a cornerstone of scientific advancement. However, traditional machine learning approaches often struggle with the large amounts of computer-generated data required, especially when dealing with complex systems like those found in multiscale modeling or stochastic simulations. In these scenarios, we might have a reliable but computationally expensive "ground truth" model (e.g., molecular dynamics) and need to derive a simpler, coarse-grained model for practical use. This process, often involving spatial averaging or coarse-graining, is typically done manually and requires significant domain expertise. This paper aims to automate this discovery process, specifically focusing on learning equations that describe the evolution of probability density functions (PDFs) of system states.
Methods
The authors propose a machine learning strategy based on sparse regression to discover differential equations. They introduce two modes:
- Direct Equation Learning (DEL): This mode aims to discover the entire differential operator from a predefined dictionary of possible terms.
- Constrained Equation Learning (CEL): This mode leverages prior knowledge or theoretical constraints to pre-define some parts of the operator, focusing the learning on discovering only the unknown "closure" terms. This is particularly useful when deriving PDF equations, as theoretical properties can significantly constrain the search space.
The core of the method involves:
- Data Generation: Simulating system dynamics using a high-fidelity model (e.g., Monte Carlo simulations of a PDE with random inputs).
- PDF/CDF Estimation: Estimating the probability density function (PDF) or cumulative distribution function (CDF) of the system state from the simulation data, often using techniques like Kernel Density Estimation (KDE).
- Dictionary Construction: Creating a dictionary of plausible terms for the differential operator, guided by theoretical considerations (e.g., the Pawula theorem for PDFs).
- Sparse Regression: Using L1 regularization and recursive feature elimination (RFE) to identify the most relevant terms and their coefficients in the differential operator, leading to a parsimonious and interpretable equation.
Datasets
The paper uses simulated data generated from solving nonlinear partial differential equations (PDEs) with uncertain parameters and/or initial conditions. These include:
- A nonlinear advection-reaction PDE with additive noise.
- A nonlinear advection-reaction PDE with multiplicative noise.
- A nonlinear conservation law (inviscid Burgers equation) with a shock.
These datasets are generated via Monte Carlo simulations, and the authors discuss the impact of the number of realizations (NMC) and mesh size on the learning process.
Key Insights and Ideas
- PDF/CDF Equations are Advantageous: Learning equations for PDFs or CDFs is more tractable than for the raw system state because these equations are linear and satisfy strong theoretical constraints, significantly reducing the search space for potential terms.
- Physics-Informed Dictionaries are Crucial: Incorporating theoretical knowledge (like the Pawula theorem) into the construction of the dictionary of possible terms is vital for successful and efficient equation discovery. This guides the machine learning model towards physically plausible solutions.
- Observable Choice Matters: The choice of what to learn (PDF vs. CDF) can significantly impact the robustness and accuracy of the discovered equation. In their experiments, learning the CDF proved more robust.
- Constrained Learning (CEL) Outperforms: When prior knowledge is available to constrain parts of the differential operator, the Constrained Equation Learning (CEL) approach generally outperforms Direct Equation Learning (DEL) in terms of accuracy, highlighting the benefit of physics-informed hypotheses.
- Interpretability and Sparsity: The use of sparse regression techniques (L1 regularization, RFE) is key to discovering interpretable equations with a minimal set of terms, balancing model complexity and predictive power.
Key Results
The authors demonstrate their method's ability to:
- Accurately reconstruct known PDF equations: In the case of the nonlinear advection-reaction PDE with additive noise, their method recovered the exact PDF equation with high accuracy, estimating coefficients with less than 5% error.
- Discover plausible closure approximations: For more complex systems where exact PDF equations are unknown, the method identified relevant terms and coefficients that represent physically meaningful closure relations.
- Handle nonlinearities and noise: The framework successfully learned equations for systems with nonlinear dynamics and stochastic inputs.
- Outperform DEL with CEL: In several experiments, CEL achieved lower cross-validation errors compared to DEL, especially when dealing with more complex systems or when physics-informed constraints were applied. For instance, in the advection-reaction PDE with multiplicative noise, CEL outperformed DEL across a wide range of RFE thresholds.
- Learn CDF equations robustly: The discovery of the CDF equation for the inviscid Burgers equation was shown to be more robust, achieving near-exact recovery with appropriate hyper-parameter tuning.
Limitations and Future Work
- Generalization Bounds: The authors acknowledge that while generalization is the goal, models are always bounded by the data they are trained on.
- Nonlocality: The current framework primarily focuses on local differential operators. Future work will explore incorporating integro-differential or fractional-derivative terms to account for nonlocality.
- Computational Efficiency: Further investigation into optimizing the algorithm's accuracy and computational efficiency is needed, including exploring relationships between numerical schemes for derivative approximation and learning accuracy, and advanced optimization techniques for hyper-parameters.
- Time Dependence: The current approach often assumes time-independent coefficients. Future studies will investigate explicit treatment of time dependence.
- Shock Handling: For systems that develop shocks, specialized treatment is required, and the paper suggests future work in this area, including learning CDF equations in the frequency domain or using finite volume methods.
- Uniqueness of Solutions: The paper notes that the discovered equations might not be unique, and the choice of observable and constraints can influence the outcome.
Summary updated