GTApprox: Surrogate modeling for industrial design

Mikhail Belyaev, Evgeny Burnaev, Ermek Kapushev, Maxim Panov, Pavel V. Prikhodko, Dmitry P. Vetrov, Dmitry Yarotsky

TL;DR

This paper introduces GTApprox, a novel software tool designed for medium-scale surrogate modeling in industrial design. GTApprox addresses limitations in existing tools by offering innovative approximation algorithms, advanced automated model selection, and user-friendly options tailored for engineering applications, ultimately leading to more accurate and efficient design optimization.

What Problem Does This Paper Address?

Industrial design processes heavily rely on optimization to find the best product configurations. However, evaluating the performance of each design configuration (the "response function") can be computationally very expensive, often requiring complex simulations. Surrogate models, or approximation models, are used to create cheaper-to-evaluate "surrogates" of these response functions. While many tools exist for building these surrogates, they often suffer from:

  • Technical User Interfaces: A lack of consistency and a steep learning curve due to the need to understand the intricacies of numerous individual modeling algorithms and their parameters.
  • Engineering Specifics: A failure to account for unique characteristics of engineering problems, such as data anisotropy, the need for smooth models with available gradients, and multi-dimensional outputs.

GTApprox aims to bridge this gap by providing a more intuitive, efficient, and engineering-aware surrogate modeling solution.

Methods

GTApprox introduces several key innovations:

  • Novel Approximation Algorithms: The tool incorporates new algorithms, including:
  • Tensor Products of Approximations (TA) and incomplete TA (iTA): These techniques are designed to handle anisotropic data by breaking down the problem into smaller, more manageable parts, each handled by a specialized approximation method. iTA is particularly useful for non-Cartesian or incomplete experimental designs.
  • Mixture of Approximations (MoA): This method addresses highly inhomogeneous response functions by partitioning the design space and building separate models for each region, ensuring better coverage of complex behaviors.
  • Gaussian Processes (GP) and Sparse Gaussian Processes (SGP): These are flexible nonlinear techniques that provide local accuracy estimates, crucial for optimization.
  • High Dimensional Approximation (HDA) and HDAGP: These adaptive techniques handle high-dimensional inputs by decomposing them into linear and nonlinear base functions.
  • Automated Model Selection: GTApprox offers two meta-algorithms to automatically choose the best technique for a given problem:
  • "Decision Tree": A rule-based system that uses data properties (size, dimensionality) and user-specified options to select a technique.
  • "Smart Selection": A more sophisticated approach that performs a numerical optimization of technique parameters by minimizing cross-validation error, using surrogate-based optimization (SBO).
  • User-Friendly Options: GTApprox provides "non-technical" options like "Accelerator" (to control training time vs. accuracy) and "Accuracy Evaluation" (to estimate model uncertainty), making it easier for engineers to express their needs without deep algorithmic knowledge.
  • Built-in Gradients: For most techniques, GTApprox automatically computes gradients (or Jacobian matrices for multi-component outputs), which are essential for optimization.

Datasets

The paper evaluates GTApprox on a collection of 31 small- and medium-scale problems. These include:

  • 23 problems defined by explicit mathematical formulas (e.g., Ackley, Rosenbrock functions, non-smooth and noisy functions).
  • 8 real-world datasets or results from complex simulations, borrowed from the UCI repository and the GdR Mascot-Num benchmark.

The training set sizes vary from 5 to 30,000 points, and input dimensionality ranges from 1 to 20. Testing is performed on a holdout set.

Key Insights and Ideas

  • Engineering Problems are Special: The paper strongly emphasizes that standard machine learning techniques often fall short for engineering applications due to issues like data anisotropy and the need for smooth, differentiable models. GTApprox is specifically designed to address these engineering-centric challenges.
  • Automated Selection is Key: Manually selecting and tuning surrogate models is time-consuming and requires expert knowledge. GTApprox's automated "Decision Tree" and "Smart Selection" algorithms significantly lower this barrier.
  • Accuracy vs. Speed Trade-off: GTApprox, by default, prioritizes accuracy, which can lead to longer training times compared to some other libraries. However, its "Accelerator" option allows users to tune this trade-off.
  • iTA for Non-Standard Designs: The iTA technique is particularly valuable for real-world scenarios where experimental designs are not perfectly structured (e.g., unions of grids, incomplete factorials), a common situation in engineering.
  • MoA for Inhomogeneous Functions: For problems where the response function behaves very differently across the design space, MoA provides a robust way to build accurate surrogate models.

Key Results

  • Superior Accuracy: GTApprox, especially with the "smart selection" algorithm, demonstrates superior accuracy compared to default implementations of scikit-learn, XGBoost, and GPy on a diverse set of test problems.
  • Competitive Training Times: While GTApprox can take longer to train, its training times remain "moderate" for medium-scale problems, and the "Accelerator" option can reduce this further.
  • Practical Applications: The paper showcases successful applications in aerospace engineering for optimizing composite stiffened panels and estimating helicopter loads, highlighting the practical benefits of GTApprox's features like automated selection and gradient availability. For instance, in the aerodynamic problem, iTA was significantly faster (10 seconds) than a conventional GP model (1800 seconds) for a 2026-point set.

Limitations and Future Work

  • Training Time: The "smart selection" algorithm can be time-consuming due to repeated surrogate model constructions. While hints can speed this up, it's a trade-off with accuracy.
  • Out-of-Memory Errors: Some techniques may encounter memory limitations on very large datasets, leading to exceptions.
  • Future Work: The paper implicitly suggests that further development of meta-algorithms and exploration of new approximation techniques will continue to enhance GTApprox's capabilities. The focus on user-friendly options and addressing specific engineering needs is likely to remain a core theme.

Summary updated

Abstract

We describe GTApprox - a new tool for medium-scale surrogate modeling in industrial design. Compared to existing software, GTApprox brings several innovations: a few novel approximation algorithms, several advanced methods of automated model selection, novel options in the form of hints. We demonstrate the efficiency of GTApprox on a large collection of test problems. In addition, we describe several applications of GTApprox to real engineering problems.
BibTeX
@article{belyaev2016gtapprox,
  author = {Mikhail Belyaev and Evgeny Burnaev and Ermek Kapushev and Maxim Panov and Pavel V. Prikhodko and Dmitry P. Vetrov and Dmitry Yarotsky},
  title = {{GTApprox: Surrogate modeling for industrial design}},
  journal = {Adv. Eng. Softw.},
  year = {2016},
  doi = {10.1016/j.advengsoft.2016.09.001},
}
Figure