Benchmarking Tabular Foundation Models as Surrogates in Expensive Evolutionary Optimization

Lu Han, Jin Wang, Yuchen Li, Haoran Gu, Shulei Liu, Ziyang Shi, Wenao Lu, Handing WangarXiv 2026 · arXiv

TL;DR

TabPFN, a powerful Transformer-based tabular foundation model, shows great promise as a surrogate model for approximating complex fitness landscapes in expensive optimization problems. However, this comprehensive benchmark reveals that it is not a universal drop-in replacement for traditional models like Gaussian Processes or Radial Basis Function networks. Its success is highly problem-dependent, requiring careful alignment with data availability, landscape complexity, and the specific role it plays within the evolutionary algorithm.

What Problem Does This Paper Address?

Real-world engineering optimization tasks—such as aerodynamic shape design or circuit parameter tuning—often rely on computationally expensive computer simulations or physical experiments. To save time and resources, researchers use Surrogate-Assisted Evolutionary Algorithms (SAEAs), which deploy cheap machine learning models (surrogates) to mimic the expensive simulation landscape and guide the search.

Traditionally, these surrogates are small-scale models like Gaussian Processes (GP) or Radial Basis Function Networks (RBFN). While useful, their simple structures struggle to fit high-dimensional, highly non-linear, or complex landscapes. Recently, TabPFN (Tabular Prior-data Fitted Network) emerged as a disruptive, Transformer-based foundation model capable of highly accurate tabular predictions out-of-the-box via in-context learning. However, because evolutionary optimization involves dynamically shifting data distributions and flexible algorithmic pipelines, it remained unclear whether TabPFN could reliably replace traditional surrogates.

Fig. 2: General framework of offline and online SAEAs

Methods

The authors conduct a systematic, top-to-bottom evaluation of TabPFN v2 as a surrogate model. First, they analyze its raw predictive capabilities (accuracy, ranking correlation, and computational runtime) against GP and RBFN baselines.

Next, they integrate TabPFN directly into various state-of-the-art SAEA frameworks, testing its performance across both offline settings (where the algorithm must rely on a static, pre-collected dataset) and online settings (where the surrogate is iteratively updated with new expensive evaluations). The benchmark spans a massive variety of problem domains: single-objective, multi-objective, constrained, large-scale (1000 dimensions), combinatorial (discrete), and mixed-variable optimization.

Datasets

The evaluation utilizes a diverse mix of synthetic landscapes, standard mathematical benchmarks, and real-world problems:

  • Synthetic Functions: 2D Ellipsoid (unimodal) and Rastrigin (multimodal) functions for landscape visualization.
  • Standard Optimization Benchmarks: The widely accepted CEC 2017 suite (single-objective), CEC 2013 (large-scale optimization), and DTLZ1–DTLZ4 (multi-objective).
  • Constrained Optimization: Ten distinct benchmark problems selected from the CEC 2006 and CEC 2010 suites.
  • Combinatorial Optimization: 12 real-world high-dimensional classification datasets (e.g., Madelon, Isolet, arcene) used for a binary feature selection task.
  • Real-World Case Study: 30 voxel-based soft robot co-design tasks from the EvoGym simulator, optimizing both robot morphology and control policies.

Key Insights and Ideas

  • The Extrapolation Trap: Counterintuitively, in large-scale (1000D) optimization, TabPFN performed significantly worse than RBFN despite having better predictive accuracy. RBFN's mathematical structure causes its predictions to wildly diverge (extrapolate aggressively) away from known data points, which implicitly forces the evolutionary algorithm to explore distant, unseen regions. TabPFN, by contrast, exhibits conservative "mean-regression" under distribution shifts, stalling global exploration.
  • Catastrophic Variance Shrinkage under Constraints: When facing highly complex or irregular constraint boundaries, TabPFN suffers from severe variance shrinkage. It excessively smooths out the predicted landscape, completely blurring the sharp mathematical boundaries needed to separate feasible solutions from infeasible ones.
  • Data-Hungry In-Context Learning: TabPFN does not fine-tune parameters; it infers patterns purely from the data provided in its context window. Consequently, it struggles in data-scarce online settings with tiny, incremental updates. It truly shines when provided with a large, sufficient batch of historical data upfront.
Fig. 4: The 2D landscape predicted by the surrogate models on the Rastrigin function.

Key Results

  • Pure Prediction Superiority: On the 10D CEC 2017 benchmark, TabPFN dominated traditional models, winning 22 out of 30 tasks in prediction error (RMSE) and 23 out of 30 in ranking correlation ($\tau$).
  • Online Optimization Gains: When substituted into an online social-learning particle swarm optimization framework (MGP-SLPSO), TabPFN-SLPSO achieved a staggering 82 overall wins compared to just 5 wins for the original GP-based baseline.
  • The EvoGym Flip: In the soft robot design case study, the choice of paradigm completely flipped the winner. In the data-scarce online setting, the baseline outperformed TabPFN (20 wins vs 9). However, in the offline setting with a pre-collected dataset, TabPFN dominated (19 wins vs 9).
  • Constant Training Time: Unlike GPs and RBFNs, whose training times scale poorly with data size and dimensionality, TabPFN's data-fitting time remains nearly constant, making it highly efficient for large datasets when paired with modern GPUs.
Fig. 5: Training and Inference Time Curves for GP, RBFN, and TabPFN on the CEC 2017 F1.

Limitations and Future Work

TabPFN's primary limitations stem from its pre-trained synthetic priors. When encountering out-of-distribution landscapes—such as highly complex constraints or massive dimensional spaces—its conservative mean-regression behavior misleads basic evolutionary search mechanisms. Furthermore, its initial training overhead is higher than lightweight RBFNs in low-dimensional, small-sample settings.

The authors highlight that future work should focus on co-designing TabPFN-specific model management strategies. Instead of using standard greedy selection, algorithms should pair TabPFN with explicitly exploration-oriented infill criteria (e.g., uncertainty or diversity metrics) to counteract its conservative extrapolation bias and unlock its full potential.

Summary updated

Abstract

Surrogate-assisted evolutionary algorithms (SAEAs) are effective methods for solving expensive optimization problems (EOPs), where surrogate models replace most expensive evaluations and critically influence the final optimization results. In recent years, tabular foundation models have advanced rapidly, and the Tabular Prior-data Fitted Network (TabPFN) has been adopted as a surrogate model for EOPs due to its strong predictive capability, demonstrating promising performance. Motivated by its potential as a surrogate model in SAEAs, this work conducts a comprehensive study that combines extensive experiments with in-depth theoretical analysis to investigate the effectiveness of TabPFN. Specifically, we perform experiments across both offline and online SAEA settings, covering diverse problem scenarios such as single-objective, multi-objective, constrained, combinatorial, mixed-variable, and engineering optimization problems. In addition, we further analyze the advantages and limitations of TabPFN within SAEAs and provide practical guidelines for its application in different optimization settings. Results show that the effectiveness of TabPFN is highly problem dependent, and it cannot replace conventional surrogates universally. Overall, TabPFN should be adopted selectively according to data availability, landscape complexity, search space characteristics, and its role within the algorithm. Customized model management strategies and role-specific algorithm design are necessary to fully exploit its advantages and avoid its pitfalls.
BibTeX
@misc{han2026benchmarking,
  author = {Lu Han and Jin Wang and Yuchen Li and Haoran Gu and Shulei Liu and Ziyang Shi and Wenao Lu and Handing Wang},
  title = {{Benchmarking Tabular Foundation Models as Surrogates in Expensive Evolutionary Optimization}},
  year = {2026},
  eprint = {2609.18130},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.18130},
}
Figure