Overview
Researchers have introduced MORE (Multi-level mOlecule gRaph prE-train), a novel foundation model designed to enhance molecular representation learning. By integrating four distinct viewpoints—node, subgraph, graph, and 3D-levels—MORE develops a comprehensive understanding of chemical structures. This multi-level approach significantly reduces "catastrophic forgetting" and outperforms existing state-of-the-art methods across various chemical property prediction tasks.
The Challenge of Molecular Representation
In drug discovery and materials science, labeled data is often scarce and expensive to obtain. Self-Supervised Learning (SSL) has emerged as a solution, where models are pretrained on massive unlabeled datasets using pretext tasks before being fine-tuned for specific applications.
However, existing models often suffer from "tunnel vision," focusing exclusively on local atomic structures or 2D topology while neglecting 3D geometry and high-level chemical properties. This narrow focus leads to catastrophic forgetting, where the model loses its general molecular knowledge during fine-tuning for a specific task. The authors of MORE aimed to build a model that understands molecules from every hierarchical angle.
The MORE Framework: Multi-Level Pretext Tasks
The core innovation of MORE is its hierarchical pretraining strategy. Using a single Graph Isomorphism Network (GIN) encoder, the model is trained against four specialized decoders to capture information at different scales:
- Node-level: Predicts the atomic numbers of randomly masked atoms to learn local chemical environments.
- Subgraph-level: Predicts MACCS keys (a set of 155 structural patterns) to identify functional groups like aromatic rings.
- Graph-level: Predicts 194 molecular descriptors (e.g., molecular weight, LogP) to capture global semantic properties.
- 3D-level: Learns to predict relative distances between atoms based on computer-generated 3D conformers, capturing spatial geometry.
By balancing these four objectives, the encoder learns a robust representation that is simultaneously local, global, structural, and geometric.
Experimental Evaluation
MORE was pretrained on a subset of ZINC15 containing 2 million molecules, with scalability tests extending up to 200 million. Performance was evaluated on seven standard MoleculeNet benchmarks, including physiology (BBBP), toxicity (Tox21, ClinTox), and target binding (HIV, BACE).
Key Results
- Linear Probing: MORE achieved an average ROC-AUC of 68.24%, significantly outperforming the second-best baseline, Infomax (63.33%).
- Full Fine-tuning: Even when the entire model was updated, MORE maintained its lead with an average ROC-AUC of 73.40%.
- Scalability: Performance improved consistently as pretraining data increased from 10M to 200M molecules, demonstrating a reliable scaling law compared to fluctuating baselines.
Key Insights
A significant contribution of the MORE paper is the quantification of "forgetting." By comparing performance between Linear Probing (frozen encoder) and Full Fine-tuning, the authors found that MORE exhibited the smallest performance gap. This suggests that its pretrained features are inherently more generalizable and require less internal restructuring for downstream tasks.
Furthermore, "leave-one-out" ablation studies revealed that different chemical tasks prioritize different levels of information. For instance, blood-brain barrier penetration (BBBP) relies heavily on global graph-level descriptors, whereas protein binding (BACE) requires fine-grained node-level details. MORE’s success stems from its ability to provide all these features simultaneously.
Conclusion and Future Work
MORE represents a significant step toward robust, multi-faceted molecular foundation models. While currently reliant on classical chemoinformatics (RDKit) for 3D ground truth, the multi-level pretext strategy is highly effective. Future research may explore applying this hierarchical approach to Transformer architectures and even larger datasets to further push the boundaries of molecular representation.
Summary updated