MolLM : A Unified Language Model for Integrating Biomedical Text with 2D and 3D Molecular Representations

Xiangru Tang, Andrew Tran, Jeffrey Too Chuan Tan, Mark GersteinbioRxiv 2023 · DOI

TL;DR

This paper introduces MolLM, a novel language model that unifies 2D and 3D molecular structures with biomedical text. By leveraging a multimodal approach and contrastive learning, MolLM achieves state-of-the-art performance on tasks like molecule-text matching and property prediction, demonstrating the critical importance of incorporating 3D structural information.

What Problem Does This Paper Address?

Current deep learning models for molecular tasks often focus on either 1D/2D molecular representations or 3D structures, but rarely integrate both with textual information. This siloed approach limits their versatility and ability to capture the full picture of molecular behavior. Specifically, many models overlook the crucial 3D structural details that dictate physical interactions, while others fail to leverage the vast knowledge embedded in biomedical text. This gap hinders the development of more comprehensive and adaptable AI models for drug discovery and other chemical research.

Methods

MolLM employs a unified pre-training framework with two main components:

  • Text Transformer Encoder: This component is initialized with weights from KV-PLM and fine-tuned on biomedical text to capture textual nuances. The tokenizer's output length is doubled to accommodate longer descriptions.
  • Molecular Transformer Encoder: This encoder is designed to process both 2D and 3D molecular structures. It uses a graph Transformer architecture that integrates positional encoding with standard graph encoding to capture atom-wise structural information. Crucially, it incorporates 3D geometric data by computing spatial distances between atoms and using a Gaussian Basis Kernel for a more nuanced representation.

The model is pre-trained using contrastive learning, a self-supervised technique that aims to bring similar representations closer together and push dissimilar ones apart in a latent space. This involves two types of losses:

  • Cross-modality contrastive loss: Aligns molecule and text representations.
  • Self-contrastive loss: Aligns different augmentations of the same modality.

Data augmentation is also employed, generating four augmented versions of each molecule (node dropping, subgraph sampling, chemical transformation, substructure removal) to improve model robustness.

Datasets

The authors constructed a new dataset for self-supervised pre-training consisting of 160K molecule-text pairings.

  • Molecular Data: Sourced from PubChem, focusing on molecules with detailed "Description" fields.
  • Textual Data: Extracted from academic papers in the S2ORC corpus, specifically sentences relevant to the molecules.

For downstream tasks, the following datasets were used:

  • Cross-modality Matching: PCDes (for supervised and zero-shot), and a dataset similar to MoMu's for zero-shot.
  • Property Prediction: MoleculeNet benchmark, including datasets like BBBP, TOX21, ToxCast, SIDER, ClinTox, MUV, HIV, and BACE.
  • Molecule Captioning: CheBI-20 dataset.
  • Molecule Editing: 200 molecules sampled from ZINC.

Key Insights and Ideas

  • 3D is Crucial: The explicit inclusion of 3D molecular structures significantly improves performance across various downstream tasks, particularly in cross-modality matching and property prediction. This highlights that 3D geometry is not just an add-on but a fundamental aspect for accurate molecular understanding.
  • Multimodality Pays Off: Combining text, 2D, and 3D molecular representations in a single model leads to richer, more versatile representations that excel in diverse tasks.
  • Contrastive Learning for Alignment: Using contrastive learning effectively aligns representations from different modalities (text and molecules) and different views of the same molecule, leading to robust embeddings.
  • Dataset Scale and Quality Matter: The authors emphasize that their larger dataset with more comprehensive textual information is key to their model's success, suggesting a need for similar high-quality, multimodal datasets in the field.
  • Encoder-Centric Approach: MolLM's encoder-only architecture allows for flexible integration with existing decoders (like MolT5 for captioning) or generative models (like MoFlow for editing), showcasing its plug-and-play utility.

Key Results

  • Cross-modality Matching: MolLM-3D outperformed all baselines, achieving an M-TR@20 (Molecule-to-Text Recall@20) of 92.05% in the supervised sentence-level task. This demonstrates superior ability to retrieve relevant text for a given molecule.
  • Property Prediction: MolLM achieved an average score of 72.40 in zero-shot scenarios, outperforming MoMu (66.96). Upon fine-tuning, it showed significant improvements, e.g., increasing from 75.6% to 80.0% on TOX21.
  • Molecule Captioning: MolLM performed comparably or slightly better than baselines like MolT5 and MoMu, even without a pre-trained text decoder, highlighting the strength of its molecular encoder.
  • Molecule Editing: The model showed varied success rates depending on the prompt, with high hit ratios for making molecules insoluble (21.5%) and less drug-like (81.5%), but lower for making them more drug-like (5.0%). This indicates potential but also areas for improvement.

Limitations and Future Work

  • Dataset Limitations: While the authors created a large dataset, they acknowledge that its scale might not fully reveal MolLM's potential, especially for text-heavy tasks. Future work could involve leveraging larger language models (like GPT-4) to enrich data or improve text sampling.
  • Molecule Editing Metrics: The current evaluation metric for molecule editing primarily focuses on property changes and may not adequately capture similarity to the original molecule. Developing more robust metrics that balance property adherence with structural similarity is suggested.
  • Infeasible Molecule Generation: The molecule editing task can sometimes produce chemically infeasible molecules. Future work could incorporate validation for chemical validity, stability, or synthesis feasibility, potentially using Reinforcement Learning or human feedback.
  • Model Architecture Exploration: While Transformers are effective, exploring non-linear combinations of 2D and 3D data, beyond the linear combination used in MolLM, could enhance expressiveness.
  • Baseline Robustness: The authors note a lack of reliable baselines, especially in de novo generation tasks, which poses challenges for evaluating and comparing models.

Summary updated

Abstract

Abstract Motivation The current paradigm of deep learning models for the joint representation of molecules and text primarily relies on 1D or 2D molecular formats, neglecting significant 3D structural information that offers valuable physical insight. This narrow focus inhibits the models’ versatility and adaptability across a wide range of modalities. Conversely, the limited research focusing on explicit 3D representation tends to overlook textual data within the biomedical domain. Results We present a unified pre-trained language model, MolLM, that concurrently captures 2D and 3D molecular information alongside biomedical text. MolLM consists of a text Transformer encoder and a molecular Transformer encoder, designed to encode both 2D and 3D molecular structures. To support MolLM’s self-supervised pre-training, we constructed 160K molecule-text pairings. Employing contrastive learning as a supervisory signal for cross-modal information learning, MolLM demonstrates robust molecular representation capabilities across 4 downstream tasks, including cross-modality molecule and text matching, property prediction, captioning, and text-prompted molecular editing. Through ablation, we demonstrate that the inclusion of explicit 3D representations improves performance in these downstream tasks. Availability and implementation Our code, data, and pre-trained model weights are all available at https://github.com/gersteinlab/MolLM .
BibTeX
@misc{tang2023i,
  author = {Xiangru Tang and Andrew Tran and Jeffrey Too Chuan Tan and Mark Gerstein and Mark B. Gerstein},
  title = {{<i>MolLM</i> : A Unified Language Model for Integrating Biomedical Text with 2D and 3D Molecular Representations}},
  year = {2023},
  doi = {10.1101/2023.11.25.568656},
  howpublished = {bioRxiv},
}
Figure