Benchmarking the acceleration of materials discovery by sequential learning

Brian A. Rohr, Helge S. Stein, Dan Guevarra, Yu Wang, Joel A. Haber, Muratahan Aykol, Santosh K. Suram, John M. GregoireChemical 2020 · DOI

Abstract

Sequential learning (SL) strategies, i.e. iteratively updating a machine learning model to guide experiments, have been proposed to significantly accelerate materials discovery and research. Applications on computational datasets and a handful of optimization experiments have demonstrated the promise of SL, motivating a quantitative evaluation of its ability to accelerate materials discovery, specifically in the case of physical experiments. The benchmarking effort in the present work quantifies the performance of SL algorithms with respect to a breadth of research goals: discovery of any "good" material, discovery of all "good" materials, and discovery of a model that accurately predicts the performance of new materials. To benchmark the effectiveness of different machine learning models against these goals, we use datasets in which the performance of all materials in the search space is known from high-throughput synthesis and electrochemistry experiments. Each dataset contains all pseudo-quaternary metal oxide combinations from a set of six elements (chemical space), the performance metric chosen is the electrocatalytic activity (overpotential) for the oxygen evolution reaction (OER). A diverse set of SL schemes is tested on four chemical spaces, each containing 2121 catalysts. The presented work suggests that research can be accelerated by up to a factor of 20 compared to random acquisition in specific scenarios. The results also show that certain choices of SL models are ill-suited for a given research goal resulting in substantial deceleration compared to random acquisition methods. The results provide quantitative guidance on how to tune an SL strategy for a given research goal and demonstrate the need for a new generation of materials-aware SL algorithms to further accelerate materials discovery.

Introduction

The search space for novel materials, particularly catalysts for clean energy applications, is vast. Sequential learning (SL) offers a path forward by using machine learning to iteratively guide experimental selection. While SL has shown promise in simulations, its effectiveness in real-world experimental discovery—where goals range from finding a single high-performer to building comprehensive predictive models—requires rigorous quantification. This study benchmarks SL algorithms against high-throughput experimental data to determine actual acceleration factors (AF) compared to random search.

Methods and Datasets

The study utilizes four distinct datasets (A, B, C, and D), each representing a chemical space of six elements (e.g., Mn-Fe-Co-Ni-La-Ce). Each dataset comprises 2,121 unique pseudo-quaternary metal oxide compositions. Ground truth for every composition was established via inkjet printing synthesis and electrocatalytic testing for the oxygen evolution reaction (OER) overpotential.

Distribution of overpotentials for the four catalyst datasets (A-D) used in the benchmarking study.

Three machine learning models were evaluated:

  • Random Forest (RF): An ensemble of decision trees.
  • Gaussian Process (GP): A Bayesian approach providing formal uncertainty.
  • Linear Ensemble (LE): A "query-by-committee" method using simple linear models.

Selection was guided by the Upper Confidence Bound (UCB) acquisition function, which balances exploitation and exploration via a hyperparameter ($\lambda$).

Key Results and Discussion

The benchmarking revealed that the effectiveness of SL is highly dependent on the research objective:

  • Finding "All" vs. "Any": SL is most powerful when the goal is to identify all top-performing materials (top 1%), achieving up to a 20x acceleration factor. For finding just a single "unicorn" material, the speedup is a more modest ~3x.
  • The Risk of Deceleration: A critical finding is that poorly tuned SL strategies can be significantly slower than random sampling. Specifically, using an exploitative strategy to build a global predictive model leads to "deceleration" as the model fails to explore the broader chemical space.
  • Model Performance at Low Budgets: For very limited experimental budgets (under 40 experiments), Random Forest often outperformed more complex models like GPs, as it requires less data to begin generating useful predictions.
  • Exploration for Global Accuracy: To build an accurate global model, pure uncertainty-based exploration ($\lambda = 0$) was the only strategy that consistently outperformed random sampling.

Conclusion

Sequential learning can dramatically accelerate materials discovery, but it is not a "one-size-fits-all" solution. Researchers must align their SL strategy—specifically the choice of model and the balance of exploration vs. exploitation—with their specific scientific goals. Future advancements will likely come from "materials-aware" algorithms that incorporate chemical intuition and improved uncertainty quantification to handle the inherent noise of experimental data.

Summary updated

Abstract

Sequential learning (SL) strategies, <i>i.e.</i> iteratively updating a machine learning model to guide experiments, have been proposed to significantly accelerate materials discovery and research. Applications on computational datasets and a handful of optimization experiments have demonstrated the promise of SL, motivating a quantitative evaluation of its ability to accelerate materials discovery, specifically in the case of physical experiments. The benchmarking effort in the present work quantifies the performance of SL algorithms with respect to a breadth of research goals: discovery of any "good" material, discovery of all "good" materials, and discovery of a model that accurately predicts the performance of new materials. To benchmark the effectiveness of different machine learning models against these goals, we use datasets in which the performance of all materials in the search space is known from high-throughput synthesis and electrochemistry experiments. Each dataset contains all pseudo-quaternary metal oxide combinations from a set of six elements (chemical space), the performance metric chosen is the electrocatalytic activity (overpotential) for the oxygen evolution reaction (OER). A diverse set of SL schemes is tested on four chemical spaces, each containing 2121 catalysts. The presented work suggests that research can be accelerated by up to a factor of 20 compared to random acquisition in specific scenarios. The results also show that certain choices of SL models are ill-suited for a given research goal resulting in substantial deceleration compared to random acquisition methods. The results provide quantitative guidance on how to tune an SL strategy for a given research goal and demonstrate the need for a new generation of materials-aware SL algorithms to further accelerate materials discovery.
BibTeX
@inproceedings{rohr2020benchmarking,
  author = {Brian A. Rohr and Helge S. Stein and Dan Guevarra and Yu Wang and Joel A. Haber and Muratahan Aykol and Santosh K. Suram and John M. Gregoire},
  title = {{Benchmarking the acceleration of materials discovery by sequential learning}},
  booktitle = {Chemical},
  year = {2020},
  doi = {10.1039/c9sc05999g},
}
Figure