Abstract
Machine Learning (ML) is increasingly used to construct surrogate models for physical simulations. We take advantage of the ability to generate data using numerical simulations programs to train ML models better and achieve accuracy gain with no performance cost. We elaborate a new data sampling scheme based on Taylor approximation to reduce the error of a Deep Neural Network (DNN) when learning the solution of an ordinary differential equations (ODE) system.
Introduction
In computational physics, Deep Neural Networks (DNNs) are frequently employed as surrogate models to accelerate expensive numerical simulations. However, many physical systems—such as chemical combustion or radioactive decay—exhibit "stiff" behavior, characterized by regions where state variables change rapidly over small intervals of input space. Standard uniform sampling often fails to capture these critical transitions, leading to high maximum errors ($L_\infty$ norm) even if average performance is acceptable. This paper introduces Taylor Based Sampling (TBS), a strategy designed to concentrate training data in regions of high physical complexity identified via local derivatives.
Methodology
The core of TBS is the identification of regions where the target function is locally complex or "steeper" than average.
The Taylor Metric
We define a metric $D_\epsilon^n$ based on a Taylor expansion of the function $f$:
$$D_\epsilon^n = \sum_{k=1}^n \frac{\epsilon^k}{k!} |f^{(k)}(x)|$$
This metric serves as a proxy for local complexity by summing the magnitudes of derivatives up to order $n$. By utilizing libraries like JAX for automatic differentiation, we can compute exact derivatives directly from simulation code.
Sampling Process
- Initial Exploration: A small set of points is sampled uniformly across the physical domain.
- Derivative Computation: Exact derivatives are calculated at these points to compute $D_\epsilon^n$.
- Density Estimation: A Gaussian Mixture Model (GMM) is fitted to the distribution of $D_\epsilon^n$ values to create a probability map of high-complexity regions.
- Targeted Sampling: Additional training points are drawn from the GMM-defined distribution, focusing data collection where the physics is hardest to learn.
Experimental Results
The TBS method was evaluated using the Bateman equations, a system of ODEs modeling radioactive decay. The input space consists of initial concentrations and time, while the output represents the resulting concentrations.
Performance Gains
Compared to Basic Sampling (BS) with the same total budget of 50,000 points, TBS achieved a massive reduction in peak error:
- 45.3% reduction in $L_\infty$ (maximum) error.
- 6.6% reduction in $L_2$ (average) error.
As shown in the figure below, the most significant improvements occurred in the "stiff" regions of the ODE solution (Figure 1b), where uniform sampling typically fails to provide sufficient resolution, leading to catastrophic local failures.
Conclusion
Taylor Based Sampling provides a robust "prior" method for improving the accuracy of physical surrogate models. By leveraging the underlying physics to guide data collection before training begins, TBS effectively minimizes peak errors in critical simulation regimes. Future work will explore scaling TBS to higher-dimensional systems and integrating it with Sobolev training, where the network learns to predict derivatives directly.
Summary updated