Molformer: Motif-Based Transformer on 3D Heterogeneous Molecular Graphs

Fang Wu, Dragomir Radev, Stan Z. LiAAAI 2023 · OpenReview

Abstract

Procuring expressive molecular representations underpins AI-driven molecule design and scientific discovery. The research mainly focuses on atom-level homogeneous molecular graphs, ignoring the rich information in subgraphs or motifs. However, it has been widely accepted that substructures play a dominant role in identifying and determining molecular properties. To address such issues, we formulate heterogeneous molecular graphs (HMGs) and introduce a novel architecture to exploit both molecular motifs and 3D geometry. Precisely, we extract functional groups as motifs for small molecules and employ reinforcement learning to adaptively select quaternary amino acids as motif candidates for proteins. Then HMGs are constructed with both atom-level and motif-level nodes. To better accommodate those HMGs, we introduce a variant of the Transformer named Molformer, which adopts a heterogeneous self-attention layer to distinguish the interactions between multi-level nodes. Besides, it is also coupled with a multi-scale mechanism to capture fine-grained local patterns with increasing contextual scales. An attentive farthest point sampling algorithm is also proposed to obtain the molecular representations. We validate Molformer across a broad range of domains, including quantum chemistry, physiology, and biophysics. Extensive experiments show that Molformer outperforms or achieves the comparable performance of several state-of-the-art baselines.

Introduction

In the field of AI-driven drug discovery, the effective representation of molecular structures is a fundamental challenge. Most existing Graph Neural Networks (GNNs) treat molecules as homogeneous graphs of atoms. However, chemical behavior is often dictated by motifs—recurring substructures like carboxyl groups or benzene rings.

Current models typically either ignore these motifs or aggregate their information into atom-level features, which prevents the model from treating functional groups as distinct semantic units. Furthermore, integrating 3D geometric information with these hierarchical structures remains difficult. Molformer addresses these gaps by representing molecules as 3D Heterogeneous Molecular Graphs (HMGs), where motifs are treated as distinct nodes alongside individual atoms.

Methods

The Molformer framework introduces three primary innovations to handle the complexity of molecular structures:

Heterogeneous Molecular Graphs (HMGs)

Input graphs are constructed with both atom and motif nodes. For small molecules, motifs are predefined functional groups (e.g., hydrocarbons, halogens). For proteins, the authors employ a Reinforcement Learning (RL) motif mining technique to adaptively select the most meaningful four-amino-acid sequences for a given task.

Molformer Architecture

The architecture utilizes a variant of the Transformer designed for HMGs. Key components include:

  • Heterogeneous Self-Attention (HSA): This layer uses learnable biases to distinguish between different types of interactions: atom-to-atom, atom-to-motif, and motif-to-motif.
  • Multi-scale Mechanism: To capture both local chemical environments and global molecular shape, the model uses distance-based masking to aggregate features at varying spatial scales.
Molformer architecture overview, showing HMG construction and heterogeneous attention.

Attentive Farthest Point Sampling (AFPS)

To condense the graph into a single molecular representation (the "readout" phase), the authors propose AFPS. Unlike standard sampling that only considers spatial distance, AFPS selects nodes that are both spatially distant and "important" according to the model’s attention scores.

Results

Molformer was evaluated across seven diverse benchmarks spanning quantum chemistry (QM7, QM8), physiology (BBBP, ClinTox, BACE), and biophysics (PDBbind).

Comparative performance of Molformer against state-of-the-art baselines.

As shown in the performance table, Molformer consistently matched or exceeded state-of-the-art results. In regression tasks, it achieved a Mean Absolute Error (MAE) of 43.2 on QM7, significantly improving upon the previous best of 47.8. In classification tasks, it reached a ROC-AUC of 0.937 on ClinTox. For protein-ligand binding (PDBbind), Molformer achieved a 1.386 RMSD, outperforming 3D-specific methods like HoloProt and Schnet.

Discussion and Conclusion

The success of Molformer highlights the importance of treating motifs as "phrases" in a chemical language, providing semantic segmentation that aids the model in understanding chemical functionality. By separating motifs into distinct nodes, the model avoids the error propagation common in atom-only representations.

While currently relying on fixed lexicons for small molecules, future work may involve learning motifs directly from data and optimizing the RL-based mining for larger protein complexes. Molformer represents a significant step forward in utilizing multi-level graph construction for molecular property prediction and scientific discovery.

Summary updated

Abstract

Procuring expressive molecular representations underpins AI-driven molecule design and scientific discovery. The research mainly focuses on atom-level homogeneous molecular graphs, ignoring the rich information in subgraphs or motifs. However, it has been widely accepted that substructures play a dominant role in identifying and determining molecular properties. To address such issues, we formulate heterogeneous molecular graphs (HMGs) and introduce a novel architecture to exploit both molecular motifs and 3D geometry. Precisely, we extract functional groups as motifs for small molecules and employ reinforcement learning to adaptively select quaternary amino acids as motif candidates for proteins. Then HMGs are constructed with both atom-level and motif-level nodes. To better accommodate those HMGs, we introduce a variant of the Transformer named Molformer, which adopts a heterogeneous self-attention layer to distinguish the interactions between multi-level nodes. Besides, it is also coupled with a multi-scale mechanism to capture fine-grained local patterns with increasing contextual scales. An attentive farthest point sampling algorithm is also proposed to obtain the molecular representations. We validate Molformer across a broad range of domains, including quantum chemistry, physiology, and biophysics. Extensive experiments show that Molformer outperforms or achieves the comparable performance of several state-of-the-art baselines. Our work provides a promising way to utilize informative motifs from the perspective of multi-level graph construction. The code is available at https://github.com/smiles724/Molformer.
BibTeX
@inproceedings{wu2023molformer,
  author = {Fang Wu and Dragomir Radev and Stan Z. Li},
  title = {{Molformer: Motif-Based Transformer on 3D Heterogeneous Molecular Graphs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=6tSAumveff},
}
Figure