FineMolTex: Towards Fine-grained Molecular Graph-Text Pre-training

Yibo Li, Yuan Fang, Mengmei Zhang, Chuan Shi

Abstract

Understanding molecular structure and related knowledge is crucial for scientific research. Recent studies integrate molecular graphs with their textual descriptions to enhance molecular representation learning. However, they focus on the whole molecular graph and neglect frequently occurring subgraphs, known as motifs, which are essential for determining molecular properties. Without such fine-grained knowledge, these models struggle to generalize to unseen molecules and tasks that require motif-level insights. To bridge this gap, we propose FineMolTex, a novel Fine-grained Molecular graph-Text pre-training framework to jointly learn coarse-grained molecule-level knowledge and fine-grained motif-level knowledge. Specifically, FineMolTex consists of two pre-training tasks: a contrastive alignment task for coarse-grained matching and a masked multi-modal modeling task for fine-grained matching. In particular, the latter predicts the labels of masked motifs and words, leveraging insights from each other, thereby enabling FineMolTex to understand the fine-grained matching between motifs and words. Finally, we conduct extensive experiments across three downstream tasks, achieving up to 230% improvement in the text-based molecule editing task. Additionally, our case studies reveal that FineMolTex successfully captures fine-grained knowledge, potentially offering valuable insights for drug discovery and catalyst design.

Introduction

In the field of AI for chemistry, multi-modal models typically attempt to link molecular graphs with textual descriptions. However, most existing frameworks, such as MoleculeSTM or MoMu, rely on coarse-grained alignment, matching entire graphs to full paragraphs of text. This approach overlooks the modular nature of chemistry, where a molecule's properties are largely dictated by its motifs (functional groups like benzene rings or carboxyl groups).

Overview of the FineMolTex framework, illustrating the transition from coarse-grained molecule-level learning to fine-grained motif-level learning.

FineMolTex addresses this limitation by moving beyond "whole-molecule" alignment. By focusing on motifs, the model can generalize to unseen molecules composed of familiar parts and excel in tasks like molecule editing that require specific structural modifications.

Methodology

FineMolTex employs two primary innovations to bridge the gap between motifs and words:

1. Importance-based Selective Masking

Unlike standard models that mask tokens randomly, FineMolTex calculates an "importance score" for every motif and word. This score reflects how much a specific token contributes to the global representation of the molecule. By selectively masking the most important tokens, the model is forced to learn the critical connections—for example, predicting a specific functional group from textual context rather than relying on generic atoms.

2. Dual Pre-training Tasks

The framework utilizes a two-pronged training strategy to capture both global and local features:

  • Contrastive Alignment (Coarse-grained): Ensures that the overall graph and text embeddings are aligned in a shared vector space.
  • Masked Multimodal Modeling (Fine-grained): The model must predict the identity of a masked motif using information from the text modality (and vice-versa) through a cross-attention Transformer layer.
Detailed architecture of FineMolTex showing the Contrastive Alignment and Masked Multimodal Modeling (MMM) pipelines with selective masking.

3. Tokenization

Molecular graphs are fragmented into chemically meaningful motifs using the BRICS algorithm, while accompanying text is tokenized using SciBERT.

Experimental Setup

The model was pre-trained on PubChemSTM (281,000 pairs) and evaluated across several benchmarks:

  • Zero-shot Retrieval: Tested on DrugBank subsets to evaluate generalization to unseen chemicals.
  • Property Prediction: Evaluated on MoleculeNet (8 classification tasks).
  • Molecule Editing: Tested using 200 molecules from the ZINC dataset with 12 distinct text prompts.

Key Results and Insights

  • Superior Molecule Editing: FineMolTex achieved a relative gain of up to 230% over baselines in text-based molecule editing, particularly when prompts specified motif-level changes (e.g., "add a hydroxyl group").
  • Zero-shot Generalization: The model consistently outperformed SOTA models like MoleculeSTM across all DrugBank datasets, proving that motif-level alignment is a "universal language" for unseen data.
  • Property Prediction: On MoleculeNet, FineMolTex showed relative gains of 3.2% to 4.7% on specific tasks (SIDER, MUV, BACE), demonstrating that cross-modal pre-training enhances graph-only representations.
  • Chemical Understanding: Qualitative t-SNE visualizations confirmed that the model clusters text tokens like "poison" near toxic motifs like arsine ([AsH3]), indicating it has learned underlying chemical properties rather than just label matching.

Conclusion and Future Work

FineMolTex demonstrates that fine-grained motif-level knowledge is essential for robust molecular representation learning. While the current model focuses on 2D graphs, future work could incorporate 3D geometric structures to capture spatial chemistry. Additionally, exploring learnable fragmentation strategies beyond the BRICS algorithm may further refine the model's understanding of complex chemical features.

Summary updated

Abstract

Understanding molecular structure and related knowledge is crucial for scientific research. Recent studies integrate molecular graphs with their textual descriptions to enhance molecular representation learning. However, they focus on the whole molecular graph and neglect frequently occurring subgraphs, known as motifs,which are essential for determining molecular properties. Without such fine-grained knowledge, these models struggle to generalize to unseen molecules and tasks that require motif-level insights. To bridge this gap, we propose FineMolTex, a novel Fine-grained Molecular graph-Text pre-training framework to jointly learn coarse-grained molecule-level knowledge and fine-grained motif-level knowledge. Specifically, FineMolTex consists of two pre-training tasks: a contrastive alignment task for coarse-grained matching and a masked multi-modal modeling task for fine-grained matching. In particular, the latter predicts the labels of masked motifs and words, leveraging insights from each other, thereby enabling FineMolTex to understand the fine-grained matching between motifs and words. Finally, we conduct extensive experiments across three downstream tasks, achieving up to 230% improvement in the text-based molecule editing task. Additionally, our case studies reveal that FineMolTex successfully captures fine-grained knowledge, potentially offering valuable insights for drug discovery and catalyst design.
BibTeX
@misc{li2024finemoltex,
  author = {Yibo Li and Yuan Fang and Mengmei Zhang and Chuan Shi},
  title = {{FineMolTex: Towards Fine-grained Molecular Graph-Text Pre-training}},
  year = {2024},
  eprint = {2409.14106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.14106},
}
Figure