Catalyst Energy Prediction with CatBERTa: Unveiling Feature Exploration Strategies through Large Language Models

Janghoon Ock, Chakradhar Guntuboina, Amir Barati FarimaniACS 2023 · DOI

Introduction

The discovery of efficient catalysts is a cornerstone of sustainable energy research, particularly for processes like carbon capture and hydrogen production. Traditionally, this discovery process relies on Density Functional Theory (DFT) to calculate adsorption energies—the strength with which a molecule (adsorbate) binds to a surface (catalyst). While DFT provides high accuracy, its computational cost is prohibitive for large-scale screening.

In recent years, Graph Neural Networks (GNNs) have emerged as a faster alternative, representing chemical systems as 3D atomic graphs. However, GNNs face significant hurdles: they require precise 3D coordinates (which are often unavailable during early screening) and operate as "black boxes" with limited interpretability. This paper introduces CatBERTa, a Transformer-based model that bypasses these limitations by treating catalyst property prediction as a natural language processing task.

Methodology: From Graphs to Text

CatBERTa is built upon the RoBERTa architecture, a robustly optimized version of the BERT (Bidirectional Encoder Representations from Transformers) model. The core innovation lies in representing the adsorbate-catalyst system as a sequence of text tokens rather than a geometric graph.

The researchers evaluated two primary textual representation strategies:

  1. Condensed Strings: These include the adsorbate's SMILES notation, the bulk composition of the catalyst, and the Miller indices (representing the surface orientation).
  2. Natural Language Descriptions: To provide richer context, the authors utilized LLMs (such as ChatGPT) and specialized tools like RoboCrystallographer to generate descriptive text. These descriptions include chemical nuances such as bonding types, coordination numbers, and electronegativity.

For the regression task, CatBERTa utilizes the embedding of the special <s> token, which is passed through a linear layer to predict the scalar adsorption energy.

Experimental Framework

The model was trained and evaluated using the Open Catalyst 2020 (OC20) dataset, specifically the Initial Structure to Relaxed Energy (IS2RE) task. Training was conducted on subsets of 10,000 and 100,000 samples to assess data efficiency. Performance was measured across four standard splits:

  • In-domain (ID): Systems similar to the training distribution.
  • OOD-adsorbate: Systems with unseen molecules.
  • OOD-catalyst: Systems with unseen surface materials.
  • OOD-both: Entirely novel combinations of adsorbates and catalysts.

Key Findings and Interpretability

The Power of Textual Descriptors

A significant finding was that CatBERTa does not require explicit 3D coordinates to achieve meaningful predictions. By including "primary interacting atoms"—the specific atoms where the adsorbate makes contact with the surface—the model's error rates dropped significantly. This suggests that the model can infer necessary spatial relationships from chemical identities and connectivity descriptions.

Attention and Chemical Intuition

One of the primary advantages of the Transformer architecture is the interpretability provided by self-attention mechanisms. By visualizing attention scores, the researchers observed that CatBERTa naturally learns to focus on the most chemically significant tokens. For instance, when predicting the energy for an $NH_3$ molecule, the model prioritized the nitrogen atom, demonstrating an emergent understanding of chemical bonding without explicit supervision.

Feature Overload and Latent Space

Interestingly, the study found that adding explicit bond lengths to the text descriptions actually increased prediction error, likely due to overfitting. This indicates that the model's pre-trained weights already contain a degree of "chemical intuition" regarding atomic distances, rendering precise measurements redundant. Furthermore, t-SNE visualizations of the model's latent space revealed that fine-tuning shifts the model's focus from general catalyst types to the specific characteristics of the adsorbate, reflecting a more nuanced understanding of the adsorption process.

Results and Performance

CatBERTa achieved a Mean Absolute Error (MAE) of 0.75 eV when trained on 100,000 samples. While this trails behind state-of-the-art geometric models like GemNet-OC, it is competitive with established GNNs such as CGCNN and SchNet.

The model's most significant achievement, however, is in systematic error cancellation. In catalyst screening, researchers are often more interested in the relative energy differences between similar systems (used to construct "volcano plots") than in absolute values. CatBERTa demonstrated a Subgroup Error Cancellation Ratio (SECR) of 19.3% in the challenging "OOD-both" split, significantly outperforming traditional GNNs. This suggests that CatBERTa is exceptionally robust for comparative chemical analysis.

Conclusion and Future Directions

CatBERTa demonstrates that text-based Large Language Models are a viable and interpretable alternative to geometric models in computational chemistry. While it does not yet surpass the absolute accuracy of the most advanced GNNs, its superior error cancellation and reduced reliance on 3D data make it a powerful tool for early-stage catalyst screening.

Future research will focus on scaling the model to larger OC20 datasets, automating the generation of scientifically grounded descriptions, and exploring hybrid architectures that combine textual intuition with geometric precision.

Summary updated

Abstract

Efficient catalyst screening necessitates predictive models for adsorption energy, which is a key descriptor of reactivity. Prevailing methods, notably graph neural networks (GNNs), demand precise atomic coordinates for constructing graph representations, while the integration of observable attributes remains challenging. This research introduces CatBERTa, an energy prediction Transformer model that uses textual inputs. Built on a Transformer encoder pretrained for language modeling purposes, CatBERTa processes human-interpretable text, incorporating target features. Attention score analysis reveals CatBERTa’s focus on tokens related to adsorbates, bulk composition, and their interacting atoms. Moreover, interacting atoms emerge as effective descriptors for adsorption configurations, while factors such as the bond length and atomic properties of these atoms offer limited predictive contributions. In predicting the adsorption energy from textual representations of initial structures, CatBERTa exhibits a precision comparable to that of conventional GNNs. Notably, in subsets recognized for their high accuracy with GNNs, CatBERTa consistently achieves a mean absolute error of 0.35 eV. Furthermore, the subtraction of the CatBERTa-predicted energies effectively cancels out their systematic errors by as much as 19.3% for chemically similar systems, surpassing the error reduction observed in GNNs. This outcome highlights its potential to enhance the accuracy of the energy difference predictions. This research establishes a fundamental framework for text-based catalyst property prediction without relying on graph representations while also unveiling intricate feature–property relationships.
BibTeX
@inproceedings{ock2023catalyst,
  author = {Janghoon Ock and Chakradhar Guntuboina and Amir Barati Farimani},
  title = {{Catalyst Energy Prediction with CatBERTa: Unveiling Feature Exploration Strategies through Large Language Models}},
  booktitle = {ACS},
  year = {2023},
  doi = {10.1021/acscatal.3c04956},
}

Loading related content...

Figure