Out-of-the-Box Tabular Foundation Models for Bayesian Optimization in Materials Discovery

Klemens Flöge, Simon Bing, Leo Grinsztajn, Eliott Kalfon, Lennart Purucker, Noah Hollmann, Frank Hutter

TL;DR

This paper demonstrates that Tabular Foundation Models (TFMs), like TabPFN, can serve as powerful "out-of-the-box" surrogate models for Bayesian Optimization in materials science. By leveraging large-scale pre-training, these models match or exceed the performance of traditional Gaussian Processes without the need for manual, task-specific kernel selection or hyperparameter tuning.

What Problem Does This Paper Address?

Materials discovery is a high-stakes game of "hot or cold." Experimental cycles are costly and slow, making Bayesian Optimization (BO) a vital tool for navigating the vast space of possible chemical compositions and processing conditions.

The heart of BO is the surrogate model—a mathematical stand-in that predicts experimental outcomes and quantifies uncertainty. For decades, Gaussian Processes (GPs) have been the de-facto choice. However, GPs come with a significant "expert tax": practitioners must choose a specific kernel (a function defining data similarity) for every new task. Pick the wrong kernel, and your optimization might stall. This paper explores whether new Tabular Foundation Models (TFMs) can act as a "drop-in" replacement that works across different domains without any manual tuning.

Methods

The authors evaluate two state-of-the-art TFMs: TabPFN and TabICL. Unlike traditional models that are trained on a specific dataset, these foundation models are pre-trained on massive synthetic datasets to perform in-context learning. When given a few experimental data points as "context," they can immediately predict the properties of new candidate materials.

The study uses a retrospective, pool-based BO setup, illustrated in Figure 1:

Figure 1: Retrospective, pool-based BO. a, candidate conditions in the experimental design space. b, the conditions the original campaign measured form the pool. c, the optimizer sees the features x of every candidate; outcomes y stay hidden until queried. d, each round, the surrogate is fit on the queried rows and scores the rest; the acquisition rule picks the next candidate, and its recorded outcome is revealed.

The researchers compared 56 different "arms"—combinations of 11 surrogate models (including 7 different GP kernels, Random Forests, and CatBoost) and 5 standard acquisition functions (like Expected Improvement and UCB). Crucially, they used the TFMs' native predictive distributions rather than forcing a Gaussian approximation, allowing the models to express complex, non-Gaussian uncertainty.

Datasets

The models were tested on 15 retrospective campaigns from two established materials science benchmarks:

  • liang: 5 tasks involving materials discovery (e.g., AgNP synthesis, Perovskite stability) with 3–5 variables and up to 600 candidates.
  • olympus: 10 tasks in chemistry and colorimetry (e.g., Suzuki reaction yield, HPLC peak area) with 3–6 variables and up to 1,386 candidates.

Key Insights and Ideas

  • The "Kernel Headache" is Real: Figure 3 highlights the fragility of Gaussian Processes. A kernel like the MLP (arccosine) might be the top performer for one dataset (AgNP) but the absolute worst for another (AutoAM). TFMs eliminate this model selection problem by providing a robust, general-purpose prior.
  • Zero-Shot Generalization: Despite being trained on synthetic data with no specific knowledge of chemistry or physics, TFMs generalized remarkably well to real-world materials data. This suggests they have captured fundamental patterns common to structured tabular data.
  • Immediate Competitiveness: Previous studies suggested TFMs might need a large initial batch of data to "catch up" to GPs. This paper finds the opposite: the best TabPFN configurations were ahead of the best GP arms even at the smallest budgets (5% of the pool).
Figure 3: GP kernel choice per dataset on liang. All arms under UCB1, the acquisition rule of the best-ranked GP arm on liang (Figure D.3). Each mark summarizes one arm’s Top-5% found over eight log-spaced budgets from 5-50% of the pool, on the raw percentage scale. Grey bars are a 2.5-97.5 percentile bootstrap over seeds. We report the analogous results on the olympus benchmark in Figure D.8.

Key Results

  • TabPFN took the top spot on both benchmarks when ranked by "Normalized Improvement" (a metric that aggregates performance across different tasks and budgets).
  • Robustness across budgets: As shown in Figure 2, TFMs (the blue and red lines) consistently outperformed or matched the best-tuned versions of traditional baselines like Random Forests and CatBoost across the entire experimental budget.
  • Low Friction: The primary advantage is "low-friction" optimization. A researcher can use TabPFN as a black-box surrogate and expect state-of-the-art results without needing a PhD in Gaussian Process covariance functions.
Figure 2: Normalized improvement (NI) against budget, best arm per surrogate family. (a) liang, Top-5%. (b) olympus, best value. Bands are the 2.5-97.5 percentile bootstrap over campaigns; seed-level differences within each campaign are analyzed in Appendix D.4. TFMs are competitive across benchmarks and budgets.

Limitations and Future Work

  • Retrospective vs. Live: The study relies on "replaying" old experiments. While this is standard for benchmarking, it doesn't perfectly capture how a model might influence a "live" closed-loop autonomous lab.
  • Dimensionality: The benchmarks used here are relatively low-dimensional (3–6 variables). Future work is needed to see if TFMs maintain their edge in high-dimensional engineering design spaces.
  • Domain-Specific Pre-training: The authors used "off-the-shelf" models. They suggest that encoding specific chemical or physical domain knowledge into the TFM's synthetic training priors could lead to even more powerful surrogates.

Summary updated

Abstract

Bayesian optimization is widely used in active learning loops for materials science. Gaussian processes serve as the de-facto surrogate model family, but require task-dependent kernel choices and can lack the modeling capacity for complex relationships. Tabular foundation models (TFMs) have recently emerged as the state-of-the-art for predictive tasks on structured data, without requiring bespoke kernels or hyperparameter tuning. We evaluate two TFMs, TabPFN and TabICL, as out-of-the-box surrogate models on two standard materials science benchmarks, comparing against a large field of baselines encompassing Gaussian processes, random forests, and CatBoost. Our findings indicate that TFMs are highly competitive surrogate models across the two benchmarks, without task-specific training for active learning or GP kernel selection. Our results motivate developing TFMs further as surrogates for active learning in materials discovery.
BibTeX
@misc{floge2026out,
  author = {Klemens Flöge and Simon Bing and Leo Grinsztajn and Eliott Kalfon and Lennart Purucker and Noah Hollmann and Frank Hutter},
  title = {{Out-of-the-Box Tabular Foundation Models for Bayesian Optimization in Materials Discovery}},
  year = {2026},
  eprint = {2611.12491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12491},
}
Figure