Uni-Mol: A Universal 3D Molecular Representation Learning Framework

Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, Guolin Ke

TL;DR

Uni-Mol is a universal 3D molecular representation learning framework that directly utilizes 3D spatial coordinates as both inputs and outputs. By pretraining on over 200 million molecular conformations and 3 million protein pockets, it achieves state-of-the-art results across 14 out of 15 property prediction benchmarks and significantly outperforms traditional tools in 3D tasks like protein-ligand binding pose prediction.

Overview of the Uni-Mol framework

What Problem Does This Paper Address?

In the world of drug discovery, a molecule's function is dictated by its 3D shape and how it fits into biological targets. However, most current Molecular Representation Learning (MRL) methods treat molecules as 1D strings (like SMILES) or 2D graphs. While these abstractions are computationally efficient, they throw away critical geometric information.

Existing 3D-aware models often face two hurdles: they are either too slow for large-scale pretraining due to complex equivariant architectures, or they are "read-only," meaning they can incorporate 3D data but cannot predict or generate new 3D structures (like a molecule's preferred shape or how it binds to a protein). Uni-Mol aims to be a "universal" framework that is efficient, 3D-native, and capable of both prediction and generation.

Methods

The authors built Uni-Mol around three core pillars:

Uni-Mol Backbone and Pretraining Architecture
  1. The Backbone: Instead of a standard Graph Neural Network (GNN), they use a modified Transformer. To make it 3D-aware, they introduce a "pair representation" initialized by Euclidean distances between atoms, which acts as a bias in the self-attention mechanism. This allows the model to capture long-range interactions that locally-connected GNNs often miss.
  2. SE(3)-Equivariant Head: To enable the model to output 3D coordinates, they designed a specific head that predicts "delta positions." This head ensures that if you rotate the input molecule, the predicted output rotates accordingly (rotational equivariance).
  3. Pretraining Strategy: They trained two separate models (a Molecule model and a Pocket model) using two self-supervised tasks:
  4. 3D Position Recovery: The model is given a molecule with "noisy" (corrupted) atom positions and must learn to recover the original 3D structure.
  5. Masked Atom Prediction: Similar to BERT in NLP, the model predicts the identity of atoms that have been hidden.

For complex tasks like protein-ligand binding, Uni-Mol uses a unique optimization approach. It predicts a distance matrix between the ligand and the protein pocket and then uses backpropagation to "move" the ligand atoms into the position that best fits that predicted matrix—a process 100x faster than traditional docking software.

Datasets

The scale of data used for pretraining is a major highlight of this work:

  • Molecular Pretraining: 209 million 3D conformations derived from 19 million unique molecules.
  • Pocket Pretraining: 3.2 million candidate protein pockets extracted from the Protein Data Bank (PDB).
  • Downstream Evaluation:
    • MoleculeNet: 15 datasets for property prediction (e.g., toxicity, solubility).
  • GEOM-QM9 & GEOM-Drugs: For molecular conformation generation.
  • CASF-2016: A gold-standard benchmark for protein-ligand docking and binding pose prediction.
  • NRDLD: For pocket druggability prediction.

Key Insights and Ideas

  • 3D is the "Ground Truth": The authors argue that 1D and 2D representations are just approximations. By training directly on 3D coordinates, the model learns the actual physical constraints of chemistry.
  • Transformers > GNNs for Molecules: While GNNs are popular, the "all-to-all" connectivity of Transformers is better suited for capturing non-bonded interactions and long-range spatial relationships in 3D space.
  • Denoising as a Proxy for Physics: The "Position Recovery" task effectively teaches the model the "force fields" of chemistry—learning which atom arrangements are physically stable and which are not.
  • Unified Ligand-Pocket Learning: By pretraining on both molecules and protein pockets using the same architecture, the framework can effectively "speak the language" of both sides of a biological interaction.

Key Results

  • Property Prediction: Uni-Mol outperformed previous state-of-the-art (SOTA) models on 14 out of 15 MoleculeNet classification and regression tasks.
  • Binding Pose Prediction: On the CASF-2016 benchmark, Uni-Mol predicted 80.35% of binding poses within a high-accuracy threshold (RMSD $\leq$ 2.0Å), a 22.58% improvement over popular traditional docking methods like Autodock Vina.
  • Efficiency: For binding pose prediction, Uni-Mol is roughly 250x faster than traditional software like GLIDE, taking only 0.2 seconds per ligand.
  • Conformation Generation: It achieved SOTA performance in generating diverse and accurate 3D shapes for molecules, outperforming dedicated generative models.

Limitations and Future Work

The authors identify several areas for growth:

  • Interaction Mechanisms: The current way the molecule and pocket models interact during finetuning is relatively simple; more sophisticated cross-attention mechanisms could improve binding predictions.
  • Model Scaling: Like LLMs in NLP, the authors suspect that even larger versions of Uni-Mol trained on more data would yield further performance gains.
  • Blind Docking: Currently, the model performs "Targeted Docking" (it needs to be told where the pocket is). Future iterations aim to tackle "Blind Docking," where the model must find the binding site on the whole protein surface itself.
  • Data Quality: There is a persistent need for higher-quality experimental 3D data, as many current benchmarks rely on computationally generated "silver standard" structures.

Summary updated

Abstract

Molecular representation learning (MRL) has gained tremendous attention due to its critical role in learning from limited supervised data for applications like drug design. In most MRL methods, molecules are treated as 1D sequential tokens or 2D topology graphs, limiting their ability to incorporate 3D information for downstream tasks and, in particular, making it almost impossible for 3D geometry prediction/generation. In this paper, we propose a universal 3D MRL framework, called Uni-Mol, that significantly enlarges the representation ability and application scope of MRL schemes. Uni-Mol contains two pretrained models with the same SE(3) Transformer architecture: a molecular model pretrained by 209M molecular conformations; a pocket model pretrained by 3M candidate protein pocket data. Besides, Uni-Mol contains several finetuning strategies to apply the pretrained models to various downstream tasks. By properly incorporating 3D information, Uni-Mol outperforms SOTA in 14/15 molecular property prediction tasks. Moreover, Uni-Mol achieves superior performance in 3D spatial tasks, including protein-ligand binding pose prediction, molecular conformation generation, etc. The code, model, and data are made publicly available at https://github.com/dptech-corp/Uni-Mol.
BibTeX
@inproceedings{zhou2023uni,
  author = {Gengmo Zhou and Zhifeng Gao and Qiankun Ding and Hang Zheng and Hongteng Xu and Zhewei Wei and Linfeng Zhang and Guolin Ke},
  title = {{Uni-Mol: A Universal 3D Molecular Representation Learning Framework}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=6K2RM6wVqKu},
}
Figure