Could Chemical Language Models benefit from Message Passing

Jiaqing Xie, Ziheng ChiACL 2024

TL;DR

This paper explores how to best combine the strengths of language models (LMs) and message-passing neural networks (MPNNs) for molecular property prediction. They find that integrating these models can improve performance on smaller molecular datasets, but the specific integration strategy matters, with contrastive learning showing promise.

What Problem Does This Paper Address?

The field of molecular machine learning is rapidly advancing, with two dominant approaches: large language models (LLMs) adapted for molecular representations (like SMILES strings) and message-passing neural networks (MPNNs) that directly leverage molecular graph structures. While both have shown success, there's a gap in understanding how to effectively combine their complementary strengths. LLMs excel at capturing sequential patterns and semantic relationships, while MPNNs are adept at encoding the inherent structural and geometric information of molecules. This paper investigates whether integrating these two paradigms can lead to better molecular property prediction, particularly by exploring how structural information from MPNNs can enhance LM performance, and vice-versa.

Methods

The authors propose two main categories of integration techniques:

  • Contrastive Learning-Based Methods: Here, an MPNN acts as an auxiliary model to supervise the training of the language model. The goal is to learn representations where similar molecules are closer in the embedding space. They explore both:
  • Node-Level Contrastive Learning: This focuses on learning embeddings for individual atoms within a molecule, aiming to make similar atoms (based on their context and neighbors) have similar embeddings.
  • Graph-Level Contrastive Learning: This approach compares the overall graph embeddings from the LM and MPNN, aiming to capture broader structural similarities.
  • Fusion-Based Methods: These methods directly combine information from both the LM and MPNN to generate outputs for downstream tasks. They consider:
  • Late Fusion: Embeddings from the LM and MPNN are extracted separately and then combined (e.g., through concatenation, summation, or other operations) before being fed into a prediction head.
  • Joint Fusion: This involves more intricate interactions, either by using MPNN outputs to fine-tune LM embeddings (MPNN2LM) or by injecting LM token embeddings into the MPNN's message-passing steps (LM2MPNN).

Datasets

The paper uses several datasets from the MoleculeNet benchmark for evaluation:

  • Small-scale datasets: HIV, BACE, ESOL, and BBBP. These are used to assess the models' performance on smaller molecular graphs.
  • Large-scale datasets: QM9. This dataset is used to evaluate performance on larger molecular graphs.

The authors note that for future work, they aim to test their methods on even larger benchmarks like PCQM4Mv2.

Key Insights and Ideas

  • Small vs. Large Graphs: The most significant finding is that integration strategies (contrastive learning and fusion) generally outperform baseline models on small-scale molecular graphs. However, these improvements diminish or disappear on larger-scale graphs. This suggests that for complex, large molecules, the inherent information in LMs or MPNNs might be sufficient, or the integration methods struggle to scale effectively.
  • Contrastive Learning's Promise: Node-level contrastive learning, in particular, shows robust performance across various tasks on smaller datasets, often outperforming other integration methods. This highlights the benefit of explicitly teaching the models to distinguish between similar and dissimilar molecular structures at a fine-grained level.
  • LM Dominance: Purely using MPNNs is generally inferior to language models for molecular property prediction, a phenomenon also observed in prior research. This underscores the power of pre-trained language models.
  • Fusion Strategy Matters: The choice of fusion operation (e.g., sum, max, concatenation) can significantly impact performance, with concatenation and max often yielding better results for late fusion.
  • Directionality of Information Flow: Injecting structural information from MPNNs into LMs (LM2MPNN) can improve MPNN performance. Conversely, injecting LM information into MPNNs (MPNN2LM) can sometimes degrade the LM's performance, suggesting that the way information is integrated is crucial.

Key Results

  • Outperformance on Small Graphs: Integration methods, especially node-level contrastive learning, showed improvements of up to ~12.8% (MAE on ESOL) compared to baselines on small datasets. For example, MPNN2LM with ChemBERTa-2 improved ESOL performance by 12.8%.
  • Limited Gains on Large Graphs: On larger datasets like QM9, the improvements from integration methods were marginal or even negative, with some fusion methods performing worse than the baseline LM.
  • Dataset Split Impact: The way datasets are split for training, validation, and testing can influence performance, with an 8:1:1 split often showing the highest predictive power.
  • Fusion Operation Choice: For late fusion, concatenation and max operations outperformed sum and gate functions for BBBP and ESOL.

Limitations and Future Work

  • Scalability: The primary limitation identified is the lack of significant improvement on large-scale datasets. The authors suggest extending their work to even larger benchmarks like PCQM4Mv2.
  • Dataset Size: The authors hypothesize that the FreeSolv dataset, being very small, might have contributed to the poor performance of integration methods on that specific task, suggesting caution when fine-tuning with additional structural features on small datasets.
  • Generalizability of Fusion: While they explored different fusion operations, the authors suggest that researchers could simply concatenate token and graph embeddings.
  • Model Choice: The paper focuses on ChemBERTa and ChemBERTa-2. Future work could explore a wider range of pretrained language models and MPNN architectures.
  • Hyperparameter Sensitivity: The paper acknowledges that dataset splitting strategies and random seeds can impact results, highlighting the need for careful experimental design.

Summary updated

BibTeX
@inproceedings{xie2024could,
  author = {Jiaqing Xie and Ziheng Chi},
  title = {{Could Chemical Language Models benefit from Message Passing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}

Loading related content...

Figure