Multi-fidelity Gaussian process surrogate modeling for regression problems in physics

Kislaya Ravi, Vladyslav Fediukov, Felix Dietrich, Tobias Neckel, Fabian Buse, Michael Bergmann, Hans-Joachim Bungartz

TL;DR

High-fidelity physical simulations are often too expensive to run repeatedly, but low-fidelity versions are cheap and inaccurate. This paper demonstrates how to "chain" these models together using advanced Gaussian Process techniques—extending non-linear methods to multiple fidelity levels and introducing a new structured kernel—to create fast, accurate "digital twins" with significantly less data.

What Problem Does This Paper Address?

In fields like aerospace engineering and plasma physics, researchers rely on complex simulations to predict system behavior. However, a single high-fidelity (HF) run can take hours or days, making tasks like design optimization or uncertainty quantification—which require thousands of runs—computationally impossible.

"Surrogate models" (or emulators) solve this by training a machine learning model to mimic the simulation. The bottleneck is that training these models still requires a significant amount of expensive HF data. "Multi-fidelity" methods try to bridge this gap by using a small amount of HF data combined with a large amount of "cheap" low-fidelity (LF) data (e.g., simulations with coarser grids or simplified physics). The challenge lies in accurately modeling the complex, often non-linear relationship between these different levels of simulation.

Methods

The authors focus on Gaussian Processes (GP), a probabilistic machine learning framework that provides not just a prediction, but a measure of uncertainty.

A Gaussian Process prior (top) represents a distribution over possible functions before seeing data. The posterior (bottom) narrows that space once training points (black dots) are observed.

Key methodological contributions include:

  • Extending Non-Linear Autoregressive (NARGP) Models: Most existing non-linear multi-fidelity models are limited to two levels (LF and HF). The authors propose a "stacked" architecture that allows these models to handle three or more fidelities by cascading the outputs of one level as inputs to the next.
  • GPDFC (Composite Kernel for Delay Fusion): They improve upon "Delay Fusion" methods—which use time-delayed LF values to approximate derivatives—by introducing a structured composite kernel. This kernel is designed to mimic a Taylor expansion, helping the model better capture how the HF signal deviates from the LF signal.
  • Deep Gaussian Processes (DGP): For cases where data points aren't "nested" (i.e., you don't have LF and HF data at the exact same input locations), they utilize Deep GPs to learn hierarchical feature representations.

Datasets

The methods were evaluated on three distinct types of problems:

  1. Academic Benchmarks: Three synthetic functions representing linear scaling, non-linear transformations, and phase-shifted oscillations.
  2. Terramechanics: Modeling the traction force of a rover wheel on soft soil. This used data from the TerRA and SCM simulations (low-fidelity) and the TROLL physical testbed (high-fidelity).
  3. Plasma Physics: Simulating microturbulence in fusion reactors using the ASTRA suite, comparing a neural network surrogate (QLKNN) as the low-fidelity source against the quasi-linear solver (QLK) as high-fidelity.

Key Insights and Ideas

  • The "Manifold" Advantage: Multi-fidelity modeling works because the relationship between LF and HF data is often much smoother than the underlying physical function itself.
Figure 4: While the high-fidelity function (orange) is highly oscillatory and hard to learn, the relationship between it and the low-fidelity function (blue) forms a smooth green manifold that is much easier for the GP to approximate with few points.
  • No Universal Winner: The "best" method depends on the physics. Simple linear models (AR1) actually outperform complex non-linear ones if the relationship between fidelities is truly linear. However, for complex physics like plasma turbulence, the new GPDFC and NARGP methods are essential.
  • The Calibration Crisis: The authors found that multi-fidelity GPs are often "overconfident"—their uncertainty bands are too narrow. They show that applying Isotonic Regression as a post-processing step is vital for making the model's error estimates reliable for real-world engineering.

Key Results

  • Data Efficiency: In the terramechanics example, the non-linear multi-fidelity model (NARDGP) achieved a lower Mean Squared Error (MSE) using only 104 high-fidelity points than a standard single-fidelity GP could achieve with 2,085 points.
  • Accuracy Boost: In plasma physics simulations, the proposed GPDFC method significantly outperformed standard GPs in predicting temperature and density gradients, which are critical for stable fusion.
  • Computational Savings: Across all real-world tests, multi-fidelity methods provided a smaller prediction error for the same total computational cost compared to traditional single-fidelity modeling.

Limitations and Future Work

  • Nested Data Constraints: Standard NARGP still struggles if LF and HF data aren't collected at the same points, though Deep GPs offer a partial solution.
  • Adaptive Sampling: The current adaptive algorithm adds points one at a time. In high-dimensional spaces, this can be slow. The authors suggest exploring "batch" adaptive sampling in future work.
  • Kernel Selection: Choosing the right kernel structure still requires significant researcher intuition or "prior knowledge" about the physical system.

Summary updated

Abstract

One of the main challenges in surrogate modeling is the limited availability of data due to resource constraints associated with computationally expensive simulations. Multi-fidelity methods provide a solution by chaining models in a hierarchy with increasing fidelity, associated with lower error, but increasing cost. In this paper, we compare different multi-fidelity methods employed in constructing Gaussian process surrogates for regression. Non-linear autoregressive methods in the existing literature are primarily confined to two-fidelity models, and we extend these methods to handle more than two levels of fidelity. Additionally, we propose enhancements for an existing method incorporating delay terms by introducing a structured kernel. We demonstrate the performance of these methods across various academic and real-world scenarios. Our findings reveal that multi-fidelity methods generally have a smaller prediction error for the same computational cost as compared to the single-fidelity method, although their effectiveness varies across different scenarios.
BibTeX
@inproceedings{ravi2024multi,
  author = {Kislaya Ravi and Vladyslav Fediukov and Felix Dietrich and Tobias Neckel and Fabian Buse and Michael Bergmann and Hans-Joachim Bungartz},
  title = {{Multi-fidelity Gaussian process surrogate modeling for regression problems in physics}},
  booktitle = {Mach. Learn. Sci. Technol},
  year = {2024},
  url = {https://openreview.net/forum?id=ihJi5VjFSQ},
}

Loading related content...

Figure