A Taylor Based Sampling Scheme for Machine Learning in Computational Physics

Paul Novello, Gaël Poëtte, David Lugato, Pietro Marco CongedoarXiv 2021 · DOI · arXiv

Abstract

Machine Learning (ML) is increasingly used to construct surrogate models for physical simulations. We take advantage of the ability to generate data using numerical simulations programs to train ML models better and achieve accuracy gain with no performance cost. We elaborate a new data sampling scheme based on Taylor approximation to reduce the error of a Deep Neural Network (DNN) when learning the solution of an ordinary differential equations (ODE) system.

Introduction

In computational physics, Deep Neural Networks (DNNs) are frequently employed as surrogate models to accelerate expensive numerical simulations. However, many physical systems—such as chemical combustion or radioactive decay—exhibit "stiff" behavior, characterized by regions where state variables change rapidly over small intervals of input space. Standard uniform sampling often fails to capture these critical transitions, leading to high maximum errors ($L_\infty$ norm) even if average performance is acceptable. This paper introduces Taylor Based Sampling (TBS), a strategy designed to concentrate training data in regions of high physical complexity identified via local derivatives.

Methodology

The core of TBS is the identification of regions where the target function is locally complex or "steeper" than average.

The Taylor Metric

We define a metric $D_\epsilon^n$ based on a Taylor expansion of the function $f$:

$$D_\epsilon^n = \sum_{k=1}^n \frac{\epsilon^k}{k!} |f^{(k)}(x)|$$

This metric serves as a proxy for local complexity by summing the magnitudes of derivatives up to order $n$. By utilizing libraries like JAX for automatic differentiation, we can compute exact derivatives directly from simulation code.

Sampling Process

  1. Initial Exploration: A small set of points is sampled uniformly across the physical domain.
  2. Derivative Computation: Exact derivatives are calculated at these points to compute $D_\epsilon^n$.
  3. Density Estimation: A Gaussian Mixture Model (GMM) is fitted to the distribution of $D_\epsilon^n$ values to create a probability map of high-complexity regions.
  4. Targeted Sampling: Additional training points are drawn from the GMM-defined distribution, focusing data collection where the physics is hardest to learn.
Heatmaps showing the Taylor metric $D_\epsilon^n$ (2a) and the resulting $L_2$ error gain (2b) across the input space.

Experimental Results

The TBS method was evaluated using the Bateman equations, a system of ODEs modeling radioactive decay. The input space consists of initial concentrations and time, while the output represents the resulting concentrations.

Performance Gains

Compared to Basic Sampling (BS) with the same total budget of 50,000 points, TBS achieved a massive reduction in peak error:

  • 45.3% reduction in $L_\infty$ (maximum) error.
  • 6.6% reduction in $L_2$ (average) error.

As shown in the figure below, the most significant improvements occurred in the "stiff" regions of the ODE solution (Figure 1b), where uniform sampling typically fails to provide sufficient resolution, leading to catastrophic local failures.

Comparison of model predictions for smooth (1a) and stiff (1b) regimes. TBS (orange) significantly outperforms BS (blue) in the stiff region.

Conclusion

Taylor Based Sampling provides a robust "prior" method for improving the accuracy of physical surrogate models. By leveraging the underlying physics to guide data collection before training begins, TBS effectively minimizes peak errors in critical simulation regimes. Future work will explore scaling TBS to higher-dimensional systems and integrating it with Sobolev training, where the network learns to predict derivatives directly.

Summary updated

Abstract

Machine Learning (ML) is increasingly used to construct surrogate models for physical simulations. We take advantage of the ability to generate data using numerical simulations programs to train ML models better and achieve accuracy gain with no performance cost. We elaborate a new data sampling scheme based on Taylor approximation to reduce the error of a Deep Neural Network (DNN) when learning the solution of an ordinary differential equations (ODE) system.
BibTeX
@misc{novello2021taylor,
  author = {Paul Novello and Gaël Poëtte and David Lugato and Pietro Marco Congedo},
  title = {{A Taylor Based Sampling Scheme for Machine Learning in Computational Physics}},
  year = {2021},
  eprint = {2101.11105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.11105},
}

Loading related content...

Figure