Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching

Shengchao Liu, Hongyu Guo, Jian Tang

Introduction

Most artificial intelligence models for drug discovery are traditionally pretrained on 2D molecular graphs, representing atoms and bonds as nodes and edges. However, a molecule’s physical and chemical properties are fundamentally dictated by its 3D geometry—specifically, how it folds and vibrates in space. While 3D data is richer, it presents significant mathematical challenges, primarily the need for models to be invariant to SE(3) transformations (rotations and translations).

This paper introduces GeoSSL-DDM, a novel self-supervised learning framework that pretrains molecular models directly on 3D geometric structures. By reframing the challenge of coordinate denoising into a task of matching pairwise atomic distances, the authors create an SE(3)-invariant proxy task that effectively captures the "potential energy surface" of molecules, leading to state-of-the-art performance across 22 downstream tasks.

Methodology: Denoising Distance Matching

The core innovation of this work is GeoSSL-DDM (Denoising Distance Matching). The method aims to maximize the Mutual Information (MI) between a stable molecular geometry and a slightly perturbed version of it.

The GeoSSL-DDM framework: extracting, perturbing, and denoising pairwise distances between atoms to learn robust geometric representations.

The process follows three primary steps:

  1. From Coordinates to Distances: Instead of predicting absolute $x, y, z$ coordinates, which are sensitive to the molecule's orientation, the model focuses on pairwise distances between atoms. Distances are inherently SE(3)-invariant, ensuring the model's predictions remain consistent regardless of spatial transformation.
  2. Score Matching via EBM: The authors utilize an Energy-Based Model (EBM) approach, treating the denoising process as a way to learn the "score" (the gradient of the log-likelihood). Mathematically, this denoising task is equivalent to learning "pseudo-forces" that push atoms back toward their stable equilibrium.
  3. Multi-level Noise: Following generative modeling techniques like diffusion, the model is trained to denoise at multiple levels of Gaussian noise, enhancing the robustness of the learned representations.

Key Insights

  • The Conformer as a Local Minimum: The authors conceptualize a molecule's stable 3D shape as a local minimum on a Potential Energy Surface (PES). Denoising teaches the model to recognize the geometric "shape" of the energy valley surrounding that minimum.
  • The Primacy of 3D Information: Ablation studies indicate that 3D coordinates are significantly more critical for predicting quantum properties than atom types alone. In many cases, 3D geometry implicitly contains the topological information usually found in 2D bond graphs.
  • Invariance through Decomposition: By decomposing coordinate denoising into pairwise distance denoising, the model respects physical laws (rotational and translational invariance) without requiring complex, computationally expensive architectural constraints.

Experimental Results

GeoSSL-DDM was pretrained on a subset of 1 million molecules from the Molecule3D dataset and evaluated against nine strong baselines across several benchmarks.

Performance comparison on the QM9 dataset. GeoSSL-DDM (ours) achieves the lowest error across all 12 quantum mechanical property prediction tasks.
  • QM9 & MD17: Using the PaiNN backbone, GeoSSL-DDM achieved state-of-the-art results on all 12 tasks in QM9 and all 8 tasks in MD17.
  • Binding Affinity: On the Atom3D benchmarks (LBA and LEP), the model significantly outperformed baselines in predicting how strongly small molecules bind to proteins—a critical task for drug discovery.
  • Backbone Agnostic: The method showed consistent improvements across different architectures, including PaiNN and SchNet, proving its versatility as a general-purpose pretraining strategy.

Conclusion and Future Work

While GeoSSL-DDM demonstrates the power of 3D geometric pretraining, the authors note that current iterations focus purely on geometry, potentially ignoring useful "shortcuts" provided by 2D bond topology. Future work may involve hybrid models that integrate both 2D and 3D signals. Additionally, the "distance matching" framework holds promise for broader applications in 3D point cloud data, such as protein folding and material science research.

Summary updated

BibTeX
@inproceedings{liu2023molecular,
  author = {Shengchao Liu and Hongyu Guo and Jian Tang},
  title = {{Molecular Geometry Pretraining with SE(3)-Invariant Denoising Distance Matching}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=CjTHVo1dvR},
}
Figure