TL;DR
This paper demonstrates that Tabular Foundation Models (TFMs), like TabPFN, can serve as powerful "out-of-the-box" surrogate models for Bayesian Optimization in materials science. By leveraging large-scale pre-training, these models match or exceed the performance of traditional Gaussian Processes without the need for manual, task-specific kernel selection or hyperparameter tuning.
What Problem Does This Paper Address?
Materials discovery is a high-stakes game of "hot or cold." Experimental cycles are costly and slow, making Bayesian Optimization (BO) a vital tool for navigating the vast space of possible chemical compositions and processing conditions.
The heart of BO is the surrogate model—a mathematical stand-in that predicts experimental outcomes and quantifies uncertainty. For decades, Gaussian Processes (GPs) have been the de-facto choice. However, GPs come with a significant "expert tax": practitioners must choose a specific kernel (a function defining data similarity) for every new task. Pick the wrong kernel, and your optimization might stall. This paper explores whether new Tabular Foundation Models (TFMs) can act as a "drop-in" replacement that works across different domains without any manual tuning.
Methods
The authors evaluate two state-of-the-art TFMs: TabPFN and TabICL. Unlike traditional models that are trained on a specific dataset, these foundation models are pre-trained on massive synthetic datasets to perform in-context learning. When given a few experimental data points as "context," they can immediately predict the properties of new candidate materials.
The study uses a retrospective, pool-based BO setup, illustrated in Figure 1:
The researchers compared 56 different "arms"—combinations of 11 surrogate models (including 7 different GP kernels, Random Forests, and CatBoost) and 5 standard acquisition functions (like Expected Improvement and UCB). Crucially, they used the TFMs' native predictive distributions rather than forcing a Gaussian approximation, allowing the models to express complex, non-Gaussian uncertainty.
Datasets
The models were tested on 15 retrospective campaigns from two established materials science benchmarks:
- liang: 5 tasks involving materials discovery (e.g., AgNP synthesis, Perovskite stability) with 3–5 variables and up to 600 candidates.
- olympus: 10 tasks in chemistry and colorimetry (e.g., Suzuki reaction yield, HPLC peak area) with 3–6 variables and up to 1,386 candidates.
Key Insights and Ideas
- The "Kernel Headache" is Real: Figure 3 highlights the fragility of Gaussian Processes. A kernel like the MLP (arccosine) might be the top performer for one dataset (AgNP) but the absolute worst for another (AutoAM). TFMs eliminate this model selection problem by providing a robust, general-purpose prior.
- Zero-Shot Generalization: Despite being trained on synthetic data with no specific knowledge of chemistry or physics, TFMs generalized remarkably well to real-world materials data. This suggests they have captured fundamental patterns common to structured tabular data.
- Immediate Competitiveness: Previous studies suggested TFMs might need a large initial batch of data to "catch up" to GPs. This paper finds the opposite: the best TabPFN configurations were ahead of the best GP arms even at the smallest budgets (5% of the pool).
Key Results
- TabPFN took the top spot on both benchmarks when ranked by "Normalized Improvement" (a metric that aggregates performance across different tasks and budgets).
- Robustness across budgets: As shown in Figure 2, TFMs (the blue and red lines) consistently outperformed or matched the best-tuned versions of traditional baselines like Random Forests and CatBoost across the entire experimental budget.
- Low Friction: The primary advantage is "low-friction" optimization. A researcher can use TabPFN as a black-box surrogate and expect state-of-the-art results without needing a PhD in Gaussian Process covariance functions.
Limitations and Future Work
- Retrospective vs. Live: The study relies on "replaying" old experiments. While this is standard for benchmarking, it doesn't perfectly capture how a model might influence a "live" closed-loop autonomous lab.
- Dimensionality: The benchmarks used here are relatively low-dimensional (3–6 variables). Future work is needed to see if TFMs maintain their edge in high-dimensional engineering design spaces.
- Domain-Specific Pre-training: The authors used "off-the-shelf" models. They suggest that encoding specific chemical or physical domain knowledge into the TFM's synthetic training priors could lead to even more powerful surrogates.
Summary updated