Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations

Shuaiqun Pan, Diederick Vermetten, Manuel López-Ibáñez, Thomas Bäck, Hao Wang

Abstract

Surrogate models provide efficient alternatives to computationally demanding real-world processes but often require large datasets for effective training. A promising solution to this limitation is the transfer of pre-trained surrogate models to new tasks. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models, typically assuming an affine transformation between the source and target functions. This paper extends previous research by addressing a broader range of transformations, including linear and nonlinear variations. Specifically, we consider the combination of an unknown input warping, such as one modeled by the beta cumulative distribution function, with an unspecified affine transformation. Our approach achieves transfer learning by employing a limited number of data points from the target task to optimize these transformations, minimizing empirical loss on the transfer dataset. We validate the proposed method on the widely used Black-Box Optimization Benchmark (BBOB) testbed and a real-world transfer learning task from the automobile industry. The results underscore the significant advantages of the approach, revealing that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch using the transfer dataset, particularly in data-scarce scenarios.

Problem Statement

In engineering disciplines such as automotive or aerospace design, researchers utilize surrogate models—mathematical approximations like Gaussian Process Regression (GPR)—to replace expensive computer simulations or physical experiments. However, these models are inherently data-hungry; training a high-fidelity surrogate for a new problem often requires hundreds of costly samples.

While transfer learning aims to mitigate this by reusing models from previous tasks, most existing methods assume a simple linear (affine) relationship between the "source" and "target" problems. In practice, real-world physics is rarely linear. This research addresses the gap where non-linear relationships between tasks render standard transfer learning ineffective.

Methodology: Hybrid Transformation

The proposed framework maps a source surrogate model ($\hat{f}^S$) to a target task ($f^T$) by learning a transformation of the input space. The core mapping is defined as: $$f^T(x) \approx \hat{f}^S(W\phi(x) + v)$$

Visual comparison of source and target functions alongside surrogates trained from scratch versus the proposed transfer learning approach.

The transformation consists of two primary components:

  • Non-linear Input Warping ($\phi$): The Beta Cumulative Distribution Function (CDF) is employed to warp the input space. Its smooth and monotonic properties allow for non-linear "stretching" or "squeezing" of the domain while preserving the function's underlying geometric characteristics, such as convexity.
  • Affine Transformation ($W, v$): This handles the rotation ($W$) and translation ($v$) of the domain.

To optimize these parameters with minimal data, the authors utilize Riemannian Gradient Descent for differentiable models (to maintain valid rotation matrices) and CMA-ES for non-differentiable models like Random Forests.

Experimental Evaluation

The method was rigorously tested on two benchmarks:

  1. BBOB (Black-Box Optimization Benchmark): A standard suite of 24 synthetic test functions (e.g., Sphere, Rosenbrock, Rastrigin) used to simulate various landscape complexities in 2D, 5D, and 10D.
  2. ABS-Braking Dataset: A real-world automotive dataset involving the optimization of anti-lock braking systems, representing a complex, non-linear transfer learning task.

Key Results and Insights

The results demonstrate that the proposed method is particularly effective in "low-data" regimes. When target data is extremely scarce (e.g., 20–40 samples), the transferred model consistently outperforms models built from scratch.

Performance comparison (SMAPE) across 24 BBOB functions, showing the transferred model (blue) consistently outperforming the scratch-built model (purple) in low-sample regimes.
  • Performance in Data-Scarce Scenarios: On 2D BBOB functions with only 20 samples, the transferred GPR model achieved lower Symmetric Mean Absolute Percentage Error (SMAPE) than scratch-built models in the vast majority of cases. This advantage is even more pronounced in 10D tasks, where 40 samples are insufficient for training from scratch but allow for effective model alignment.
  • Geometric Preservation: The use of smooth, monotonic warping ensures that the transfer process does not distort fundamental properties of the original model, such as its ability to identify global minima.
  • Diminishing Returns: As target data volume increases (e.g., beyond 80 samples in low dimensions), the advantage of transfer learning fades, and training from scratch eventually becomes superior. This suggests that transfer learning is most valuable for rapid initialization.
  • Real-World Impact: In the automotive braking task, the "Full" transfer (warping + affine) consistently outperformed "Affine-only" transfer, proving that non-linear warping is essential for capturing real-world engineering complexities.

Limitations and Future Work

The authors note that the method may overfit on extremely small datasets (e.g., <5 samples) where noise can lead to poor generalization. Additionally, the added parameters for non-linear warping increase optimization difficulty. Future research will explore integrating Active Learning to improve data efficiency further and investigating alternative warping functions, such as the Kumaraswamy distribution, for increased flexibility and speed.

Summary updated

Abstract

Surrogate models provide efficient alternatives to computationally demanding real world processes but often require large datasets for effective training. A promising solution to this limitation is the transfer of pre-trained surrogate models to new tasks. Previous studies have investigated the transfer of differentiable and non-differentiable surrogate models, typically assuming an affine transformation between the source and target functions. This paper extends previous research by addressing a broader range of transformations, including linear and nonlinear variations. Specifically, we consider the combination of an unknown input warping, such as one modeled by the beta cumulative distribution function, with an unspecified affine transformation. Our approach achieves transfer learning by employing a limited number of data points from the target task to optimize these transformations, minimizing empirical loss on the transfer dataset. We validate the proposed method on the widely used Black-Box Optimization Benchmark (BBOB) testbed and a real-world transfer learning task from the automobile industry. The results underscore the significant advantages of the approach, revealing that the transferred surrogate significantly outperforms both the original surrogate and the one built from scratch using the transfer dataset, particularly in data-scarce scenarios.
BibTeX
@inproceedings{pan2025transfer,
  author = {Shuaiqun Pan and Diederick Vermetten and Manuel López-Ibáñez and Thomas Bäck and Hao Wang},
  title = {{Transfer Learning of Surrogate Models: Integrating Domain Warping and Affine Transformations}},
  booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion},
  year = {2025},
  doi = {10.1145/3712255.3734291},
}
Figure