9 Kernel methods for surrogate modeling

Gabriele Santin, Bernard Haasdonk

Abstract

Kernel methods represent a powerful class of techniques for surrogate modeling, offering flexibility and strong functional analytic foundations. By utilizing kernel expansions—linear combinations of kernel functions derived from scattered input-output samples—these methods provide a meshless alternative to traditional grid-based interpolation. This approach effectively mitigates the curse of dimensionality, making it suitable for high-dimensional engineering problems. This paper explores regularized interpolation, greedy approximation, and Support Vector Regression (SVR), demonstrating their application in biomechanical simulations where they achieve significant computational speedups.

Introduction

In modern engineering, high-fidelity simulations such as the Finite Element Method (FEM) are indispensable but computationally expensive. A single simulation can take seconds or minutes, which becomes a bottleneck for tasks requiring thousands of evaluations, such as optimization or uncertainty quantification. While Model Order Reduction (MOR) can alleviate this, it often requires "white-box" access to the model's internal equations.

Kernel methods address the "black-box" scenario, where only input-output pairs are available. Unlike traditional interpolation that relies on structured grids—which suffer from exponential complexity as dimensions increase—kernel methods are inherently meshless. They operate on arbitrarily scattered data points, providing a robust framework for building fast surrogates in high-dimensional spaces.

Theoretical Framework: RKHS and the Kernel Trick

The mathematical elegance of kernel methods stems from Reproducing Kernel Hilbert Spaces (RKHS). In this framework, a kernel function $k(x, y)$ corresponds to an inner product in a high-dimensional feature space. The "Kernel Trick" allows algorithms to operate in this feature space implicitly, enabling the modeling of complex nonlinear relationships through simple linear combinations of kernel evaluations.

Surrogate Modeling Techniques

The paper focuses on three primary methodologies for constructing kernel-based surrogates:

1. Regularized Kernel Interpolation

This approach seeks a function that passes through (or near) the training data points. By solving a linear system to determine coefficients for the kernel expansion, it incorporates a regularization parameter ($\lambda$). This parameter is crucial for balancing data fidelity with smoothness, effectively handling noise and improving the numerical stability of the resulting model.

2. Vectorial Kernel Orthogonal Greedy Algorithm (VKOGA)

To ensure the surrogate is computationally efficient, sparsity is essential. VKOGA is an incremental algorithm that selects a small subset of the most informative data points to serve as kernel centers. Selection is guided by error indicators:

  • P-greedy: Uses the Power function, a data-independent upper bound on the interpolation error, ensuring a uniform error reduction across the domain.
  • f-greedy: Uses the actual residual (approximation error) to select points where the model performs poorly. The use of a Newton basis ensures numerical stability during this greedy selection process, allowing for efficient, matrix-free implementations.

3. Support Vector Regression (SVR)

SVR is a well-known machine learning technique that employs an $\epsilon$-insensitive loss function. It ignores errors smaller than a threshold $\epsilon$, which naturally results in a sparse model where only "support vectors" (points that lie on or outside the error tube) contribute to the final expansion.

Case Study: Biomechanical Spine Modeling

The effectiveness of these methods was evaluated using a complex biomechanical model of the human spine. The task involved modeling the coupling between a multibody system (MBS) and an FEM submodel of intervertebral disks.

  • Data: 1,370 samples (90% training, 10% testing).
  • Inputs: 3D vector representing spatial displacements and angular inclination of a vertebra.
  • Outputs: 3D vector representing force components and momentum.

Results and Discussion

The experimental results highlight the trade-offs between different kernel approaches:

  • Accuracy: VKOGA variants significantly outperformed SVR. Specifically, VKOGA achieved a maximum relative error of approximately $2 \times 10^{-1}$, while SVR struggled with a relative error of $1.4 \times 10^1$, primarily due to its inability to accurately capture small-magnitude output components.
  • Efficiency and Sparsity: Sparsity is the primary driver of evaluation speed. VKOGA models were not only more accurate but also trained faster (under 2 seconds) compared to SVR (nearly 60 seconds for hyperparameter tuning).
  • Computational Speedup: The most striking result is the performance gain. While the full FEM simulation required roughly 3 seconds per evaluation, the kernel surrogates performed the same task in $10^{-5}$ to $10^{-6}$ seconds. This represents a projected speedup factor of over 300,000, enabling real-time applications that were previously impossible.

Limitations and Future Directions

Despite their success, several challenges remain. The "flat limit" problem occurs with certain kernels (like the Gaussian) where the system becomes ill-conditioned as the shape parameter approaches zero. Furthermore, while greedy selection rules like f-greedy work well in practice, their theoretical convergence rates are less understood than the P-greedy rule. Future work involves better integration of these data-driven surrogates into classical numerical solvers to accelerate complex dynamical systems further.

Conclusion

Kernel methods provide a robust, meshless framework for creating high-performance surrogate models from scattered data. By leveraging RKHS theory and greedy approximation techniques like VKOGA, researchers can achieve massive computational speedups without sacrificing significant accuracy, paving the way for real-time simulation and optimization in complex engineering domains.

Summary updated

Abstract

This chapter deals with kernel methods as a special class of techniques for surrogate modeling. Kernel methods have proven to be efficient in machine learning, pattern recognition and signal analysis due to their flexibility, excellent experimental performance and elegant functional analytic background. These data-based techniques provide so called kernel-expansions, i. e., linear combinations of kernel functions which are generated from given input-output point samples that may be arbitrarily scattered. In particular, these techniques are meshless, do not require or depend on a grid, hence are less prone to the curse of dimensionality, even for highdimensional problems.
BibTeX
@misc{santin2019kernel,
  author = {Gabriele Santin and Bernard Haasdonk},
  title = {{9 Kernel methods for surrogate modeling}},
  year = {2019},
  eprint = {1907.10556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.10556},
}
Figure