GeomCLIP: Contrastive Geometry-Text Pre-training for Molecules

Teng Xiao, Chao Cui, Huaisheng Zhu, Vasant G. Honavar

Introduction

In drug discovery and materials science, a molecule's function is fundamentally dictated by its three-dimensional conformation. While massive datasets of 1D (SMILES) and 2D (graphs) representations exist, high-quality 3D geometric data linked to scientific literature is scarce. Existing multimodal models often rely on approximate 3D structures generated by fast but noisy algorithms, which lack the precision of "ground-state" geometries—the stable shapes molecules naturally assume.

GeomCLIP addresses this gap by introducing a multimodal pre-training framework that aligns high-fidelity 3D molecular geometries with biomedical text descriptions. By training on a newly curated dataset of over 200,000 ground-state structures, GeomCLIP achieves state-of-the-art performance across various molecular understanding tasks.

Methodology: The GeomCLIP Framework

GeomCLIP utilizes a dual-encoder architecture inspired by CLIP (Contrastive Language-Image Pre-training). The framework consists of a 3D Geometry Encoder (based on the Uni-Mol architecture) and a Text Encoder (utilizing Sci-BERT).

The GeomCLIP dual-encoder architecture featuring contrastive alignment and unimodal denoising.

The model is trained using a hybrid objective. Beyond standard Contrastive Alignment, which pulls embeddings of matching molecule-text pairs closer in a shared latent space, GeomCLIP incorporates Unimodal Denoising. This task requires the 3D encoder to reconstruct original atom positions from noisy versions, ensuring the model retains "geometric intuition" and physical common sense, preventing it from over-simplifying structures just to match text features.

Data: The PubChem3D Dataset

A core contribution of this work is the creation of PubChem3D, a high-quality dataset containing 203,257 pairs of ground-state 3D geometries and biomedical text. Geometries were sourced from PubChemQC and GEOM, utilizing DFT-calculated (Density Functional Theory) structures. Text annotations were retrieved from PubChem, covering properties like toxicity, solubility, and biological roles. This high-fidelity data provides a significant performance boost over models trained on approximate 3D structures.

Evaluation and Key Results

GeomCLIP was evaluated across three primary downstream categories: property prediction, retrieval, and captioning.

Downstream evaluation tasks including property prediction, retrieval, and captioning.
  • Property Prediction: On the QM9 benchmark, GeomCLIP achieved the lowest Mean Absolute Error (MAE) across all 12 quantum mechanics tasks, outperforming specialized 3D models like Uni-Mol and 3D-MoLM.
  • Zero-Shot Retrieval: In molecule-to-text and text-to-molecule retrieval, GeomCLIP reached an accuracy of 53.50%, a significant improvement over the previous best of 51.52%.
  • Molecule Captioning: The model produced more accurate and detailed natural language descriptions of 3D molecules, reflected in superior BLEU and ROUGE scores compared to baseline models.

Conclusion and Future Work

GeomCLIP demonstrates that ground-state geometries and scientific text act as powerful mutual supervisory signals. While the current dataset of 200k pairs is a milestone, it remains small compared to general-purpose AI datasets. Future work will focus on scaling the data, modeling the dynamic ensembles of molecular movement (conformer dynamics), and integrating more diverse literature-based descriptions to further refine the model’s understanding of the molecular world.

Summary updated

Abstract

Pretraining molecular representations is crucial for drug and material discovery. Recent methods focus on learning representations from geometric structures, effectively capturing 3D position information. Yet, they overlook the rich information in biomedical texts, which detail molecules' properties and substructures. With this in mind, we set up a data collection effort for 200K pairs of ground-state geometric structures and biomedical texts, resulting in a PubChem3D dataset. Based on this dataset, we propose the GeomCLIP framework to enhance for multi-modal representation learning from molecular structures and biomedical text. During pre-training, we design two types of tasks, i.e., multimodal representation alignment and unimodal denoising pretraining, to align the 3D geometric encoder with textual information and, at the same time, preserve its original representation power. Experimental results show the effectiveness of GeomCLIP in various tasks such as molecular property prediction, zero-shot text-molecule retrieval, and 3D molecule captioning. Our code and collected dataset are available at \url{https://github.com/xiaocui3737/GeomCLIP}
BibTeX
@inproceedings{xiao2024geomclip,
  author = {Teng Xiao and Chao Cui and Huaisheng Zhu and Vasant G. Honavar},
  title = {{GeomCLIP: Contrastive Geometry-Text Pre-training for Molecules}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822346},
}
Figure