Towards Fine-grained Molecular Graph-Text Pre-training

Yibo Li, Yuan Fang, Mengmei Zhang, Chuan ShiSubmission to ICLR 2025 · Withdrawn

TL;DR

Current molecular graph-text models align descriptions to entire molecules, missing the sub-molecular functional groups—known as motifs—that dictate biochemical properties. FineMolTex introduces a dual-level pre-training framework that pairs coarse-grained contrastive learning with masked multi-modal modeling across motifs and text tokens. This fine-grained grounding enables up to a 230% improvement in text-guided molecule editing and delivers state-of-the-art results on zero-shot retrieval and property prediction.

What Problem Does This Paper Address?

Recent multimodal foundation models connect 2D molecular graphs with textual scientific descriptions (such as patent abstracts, drug leaflets, and PubMed summaries) to bypass expensive labeled property data. However, previous frameworks (e.g., MoleculeSTM, MoMu) operate almost exclusively at the whole-molecule level.

This coarse alignment overlooks motifs—frequently recurring chemical subgraphs and functional groups (like benzene rings, hydroxyl groups, or sulfonamides) that directly govern chemical reactivity, pharmacodynamics, and solubility. Consequently, whole-molecule models struggle when generalizing to novel molecules composed of familiar motifs, and fail at localized, motif-level editing tasks where an instruction requires adding, removing, or swapping a specific functional moiety.

Methods

FineMolTex jointly learns coarse- and fine-grained graph-text semantics via two complementary objectives:

[Molecule Graph] ---> BRICS Motifs ---> GraphMVP Encoder -\
                                                          +-> Cross-Attention -> Masked Token Prediction
[Text Description] -> Word Tokens ----> SciBERT Encoder --/
  1. Tokenization and Encoding:
  2. Molecular Graphs: Decomposed into motif sequences via the BRICS algorithm and BFS traversal, prefixed with a global <MOL> token. Each motif is encoded with a pre-trained Graph Isomorphism Network (GraphMVP) followed by a mean-pooling readout and topological position embeddings.
  3. Text: Tokenized using SciBERT, prefixed with <CLS>.
  4. Both modalities first pass through separate Transformer self-attention layers to encode contextual dependencies within each modality.
  5. Coarse-Grained Contrastive Alignment:
  6. Standard InfoNCE contrastive loss aligns the global representations of matching graph-text pairs (<MOL> and <CLS>) while pushing negative pairs apart.
  7. Fine-Grained Masked Multi-Modal Modeling (MMM):
  8. The model masks ~20% of intermediate-frequency motifs and ~15% of text tokens.
  9. Cross-attention layers fuse information between the graph and text sequences.
  10. Separate classification heads reconstruct the masked motifs from textual context (e.g., reconstructing a sulfonamide group given the word "propanesulfonic") and vice versa.

Datasets

  • Pre-training: PubChemSTM, containing 281,000 paired molecular graphs and descriptive texts curated from PubChem.
  • Evaluation:
  • Cross-Modal Retrieval: Zero-shot retrieval on DrugBank-Pharmacodynamics, molecule-ATC, and DrugBank-Description.
  • Text-Based Molecule Editing: 200 molecules from ZINC edited against 12 text prompts (8 property-altering prompts such as water solubility, and 4 motif-targeting prompts such as introducing chloride or benzene).
  • Single-Modality Property Prediction: Eight binary classification benchmarks from MoleculeNet (BBBP, Tox21, ToxCast, SIDER, ClinTox, MUV, HIV, and BACE).

Key Insights and Ideas

  • Motifs Enable Zero-Shot Generalization: Even when a test molecule has never been seen as a whole, its constituent motifs almost certainly have been. Aligning text tokens directly to motifs allows the model to recognize unseen compounds compositionally.
  • Cross-Modal Reconstruction as Fine-Grained Supervision: Unsupervised subgraph-text alignment is typically noisy. By forcing the network to predict masked chemical subgraphs using surrounding text hints across a cross-attention bridge, the model naturally maps chemical vocabulary (e.g., "sulf", "amine") to specific graph fragments without explicit human-annotated bounding boxes.
  • Fine-Grained Knowledge Transfers to Graph-Only Tasks: Ablations show that learning sub-molecular text-motif correspondences improves pure molecular property classification on MoleculeNet, demonstrating that multimodal motif training acts as a strong regularizer for graph representations.

Key Results

  • Text-Based Molecule Editing: FineMolTex achieved up to a 230% relative improvement in hit ratio over baselines (MoleculeSTM, MoMu, MolCA) on prompts targeting specific motifs, and consistently improved directed physical property shifts (e.g., lowering LogP for solubility).
  • Cross-Modal Retrieval: On zero-shot retrieval with $T=20$ candidate pools, FineMolTex achieved 85.80% accuracy on DrugBank-Pharmacodynamics (vs. 81.08% for MoleculeSTM) and 49.45% on molecule-ATC (vs. 44.13% for MoleculeSTM).
  • Molecular Property Prediction: Outperformed unimodal GNNs and multimodal models across MoleculeNet, achieving the highest overall average ROC-AUC score (76.4% vs. 74.6% for MoleculeSTM), including notable gains on BACE (84.6% vs. 80.8%) and ClinTox (92.7% vs. 92.5%).
  • Interpretability: t-SNE projections and LIME attribution confirm that motifs cluster tightly around their chemical synonyms and property keywords (e.g., arsenic motifs clustering around the word "poison").

Limitations and Future Work

  • Rule-Based Motif Decomposition: The framework relies on BRICS heuristics to fragment graphs, which may split rings or miss non-standard pharmacophores.
  • Restricted Masking Vocabulary: Motifs with very high or very low frequencies were excluded from masking during pre-training to stabilize training, potentially limiting comprehension of rare functional groups.
  • 3D Conformations: The current model operates primarily on 2D topological graphs; integrating 3D stereochemical conformations at the motif level remains an open direction for drug-receptor binding applications.

Summary updated

Abstract

Understanding molecular structure and related knowledge is crucial for scientific research. Recent studies integrate molecular graphs with their textual descriptions to enhance molecular representation learning. However, they focus on the whole molecular graph and neglect frequently occurring subgraphs, known as motifs, which are essential for determining molecular properties. Without such fine-grained knowledge, these models struggle to generalize to unseen molecules and tasks that require motif-level insights. To bridge this gap, we propose FineMolTex, a novel Fine-grained Molecular graph-Text pre-training framework to jointly learn coarse-grained molecule-level knowledge and fine-grained motif-level knowledge. Specifically, FineMolTex consists of two pre-training tasks: a contrastive alignment task for coarse-grained matching and a masked multi-modal modeling task for fine-grained matching. In particular, the latter predicts the labels of masked motifs and words, leveraging insights from each other, thereby enabling FineMolTex to understand the fine-grained matching between motifs and words. Finally, we conduct extensive experiments across three downstream tasks, achieving up to 230% improvement in the text-based molecule editing task. Additionally, our case studies reveal that FineMolTex successfully captures fine-grained knowledge, potentially offering valuable insights for drug discovery and catalyst design.
BibTeX
@misc{li2024towards,
  author = {Yibo Li and Yuan Fang and Mengmei Zhang and Chuan Shi},
  title = {{Towards Fine-grained Molecular Graph-Text Pre-training}},
  year = {2024},
  eprint = {2409.14106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.14106},
}

Loading related content...

Figure