Navigating the Design Space of Equivariant Diffusion-Based Generative Models for De Novo 3D Molecule Generation

Tuan Le, Julian Cremer, Frank Noé, Djork-Arné Clevert, Kristof Schütt

Abstract

Deep generative diffusion models are a promising avenue for 3D de novo molecular design in materials science and drug discovery. However, their utility is still limited by suboptimal performance on large molecular structures and limited training data. To address this gap, we explore the design space of E(3)-equivariant diffusion models, focusing on previously unexplored areas. Our extensive comparative analysis evaluates the interplay between continuous and discrete state spaces. From this investigation, we present the EQGAT-diff model, which consistently outperforms established models for the QM9 and GEOM-Drugs datasets. Significantly, EQGAT-diff takes continuous atom positions, while chemical elements and bond types are categorical and uses time-dependent loss weighting, substantially increasing training convergence, the quality of generated samples, and inference time. We also showcase that including chemically motivated additional features like hybridization states in the diffusion process enhances the validity of generated molecules. To further strengthen the applicability of diffusion models to limited training data, we investigate the transferability of EQGAT-diff trained on the large PubChem3D dataset with implicit hydrogen atoms to target different data distributions. Fine-tuning EQGAT-diff for just a few iterations shows an efficient distribution shift, further improving performance throughout data sets. Finally, we test our model on the Crossdocked data set for structure-based de novo ligand generation, underlining the importance of our findings showing state-of-the-art performance on Vina docking scores.

Introduction

Generating 3D molecules de novo is a critical challenge in drug discovery and materials science. A successful generative model must balance two primary requirements: maintaining E(3)-equivariance (ensuring internal representations respond predictably to spatial rotations and translations) and adhering to strict chemical validity rules, such as valency constraints.

Existing diffusion models often struggle with larger molecular structures and are frequently bottlenecked by limited training data. Furthermore, fundamental design choices—such as whether to treat atom types as continuous or discrete variables—have remained unresolved. This work introduces EQGAT-diff, an Equivariant Graph Attention Diffusion model that systematically optimizes these design choices to achieve superior performance in 3D molecular design.

Methodology

EQGAT-diff incorporates several key technical innovations designed to improve both efficiency and sample quality:

  • Hybrid State Space: Unlike models that treat all features as continuous, EQGAT-diff utilizes continuous coordinates for atom positions while treating chemical elements and bond types as categorical (discrete) variables. This prevents the generation of ambiguous, non-physical atom types.
  • Time-dependent Loss Weighting ($w_s(t)$): By focusing the model's training on stages with a higher signal-to-noise ratio (SNR), the authors achieved faster convergence and more stable molecule generation compared to uniform weighting.
  • Domain-Informed Features: The diffusion process directly incorporates chemical features such as hybridization states and aromaticity, providing stronger inductive biases for chemical validity.
  • Transfer Learning Strategy: To mitigate data scarcity, the model is pre-trained on the massive PubChem3D dataset using a simplified representation with implicit hydrogens, then fine-tuned on specific target distributions.

Results and Analysis

Impact of Discrete State Spaces

A central finding of this study is the superiority of discrete state spaces for categorical chemical features. As shown in the figure below, treating atom types and bonds as discrete categories leads to significantly higher molecular stability, particularly as the number of atoms in the molecule increases.

Molecule stability as a function of the number of atoms, comparing discrete and continuous state spaces.

Treating these identities as discrete prevents the model from producing "blurry" intermediate states, a common failure mode in continuous-only models. On the GEOM-Drugs dataset, EQGAT-diff achieved a molecular stability of 91.60%, outperforming previous state-of-the-art models like MiDi (89.7%).

Efficiency of Transfer Learning

The transfer learning approach proved highly effective for adapting to new data distributions with minimal additional training. By pre-training on PubChem3D and fine-tuning on target datasets, the model quickly reaches high performance levels even with small subsets of the target data.

Impact of fine-tuning on various performance metrics across different subsets of the target dataset.

As illustrated above, metrics such as validity and stability improve rapidly within the first 25% of the fine-tuning subset. This strategy allows the model to leverage large-scale general chemical knowledge while specializing in specific molecular domains.

Performance Benchmarks

EQGAT-diff established new state-of-the-art results across several benchmarks. It is approximately 2x faster to train and run than competing models like MiDi while using fewer parameters. In structure-based design tasks (Crossdocked), it produced ligands with higher Vina docking scores than specialized models like TargetDiff, demonstrating its versatility in practical drug design scenarios.

Conclusion

EQGAT-diff demonstrates that careful navigation of the diffusion model design space—specifically regarding state spaces and loss weighting—can yield significant gains in 3D molecule generation. While the model excels in validity and stability, future work will focus on integrating synthetic accessibility scores to ensure that generated molecules can be practically synthesized in a laboratory setting. Scaling these methods to larger biological systems like proteins remains an important frontier.

Summary updated

Abstract

Deep generative diffusion models are a promising avenue for 3D de novo molecular design in materials science and drug discovery. However, their utility is still limited by suboptimal performance on large molecular structures and limited training data. To address this gap, we explore the design space of E(3)-equivariant diffusion models, focusing on previously unexplored areas. Our extensive comparative analysis evaluates the interplay between continuous and discrete state spaces. From this investigation, we present the EQGAT-diff model, which consistently outperforms established models for the QM9 and GEOM-Drugs datasets. Significantly, EQGAT-diff takes continuous atom positions, while chemical elements and bond types are categorical and uses time-dependent loss weighting, substantially increasing training convergence, the quality of generated samples, and inference time. We also showcase that including chemically motivated additional features like hybridization states in the diffusion process enhances the validity of generated molecules. To further strengthen the applicability of diffusion models to limited training data, we investigate the transferability of EQGAT-diff trained on the large PubChem3D dataset with implicit hydrogen atoms to target different data distributions. Fine-tuning EQGAT-diff for just a few iterations shows an efficient distribution shift, further improving performance throughout data sets. Finally, we test our model on the Crossdocked data set for structure-based de novo ligand generation, underlining the importance of our findings showing state-of-the-art performance on Vina docking scores.
BibTeX
@inproceedings{le2024navigating,
  author = {Tuan Le and Julian Cremer and Frank Noé and Djork-Arné Clevert and Kristof Schütt},
  title = {{Navigating the Design Space of Equivariant Diffusion-Based Generative Models for De Novo 3D Molecule Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=kzGuiRXZrQ},
}
Figure