Lang2Mol-Diff: A Diffusion-Based Generative Model for Language-to-Molecule Translation Leveraging SELFIES Representation

Nguyen Doan Hieu Nguyen, Nhat Truong Pham, Duong Thanh Tran, Balachandran ManavalanACL 2024

TL;DR

This paper introduces Lang2Mol-Diff, a novel diffusion-based generative model that translates natural language descriptions into valid molecular structures. By employing the SELFIES representation and a unified BioT5 encoder, it overcomes limitations of previous autoregressive models and SMILES-based diffusion methods, achieving state-of-the-art validity in molecule generation.

What Problem Does This Paper Address?

The challenge of generating novel molecules (de novo molecule generation) from textual descriptions is crucial for drug discovery and materials science. Traditional methods struggle with the vastness of chemical space and issues like molecule validity. While recent advances have leveraged language models and diffusion techniques, existing approaches face hurdles:

  • SMILES Validity Issues: The SMILES (Simplified Molecular Input Line Entry System) representation, while widely used, can lead to invalid molecular structures when generated by models.
  • Autoregressive Model Limitations: Models that generate sequences one element at a time (autoregressive models) can propagate errors and struggle with long-range dependencies in molecular strings.
  • Diffusion Model Complexity: While diffusion models have shown promise, some, like TGM-DLM, require a multi-stage approach (generation followed by correction) which may not be optimal.

This paper aims to bridge this gap by proposing a single-stage generative model that produces valid molecules directly from text, leveraging a more robust molecular representation and a diffusion mechanism.

Methods

Lang2Mol-Diff builds upon two key innovations:

  1. SELFIES Representation: Instead of SMILES, the model uses SELFIES (SELF-referencIng Embedded Strings). SELFIES is designed to be inherently valid, meaning any SELFIES string can be decoded into a chemically valid molecule. This directly addresses the validity issues plaguing SMILES-based generation.
  2. Diffusion-Based Generative Model: The core of the method is a diffusion model, inspired by image and text generation techniques. This involves:
  3. Forward Process: Gradually adding Gaussian noise to the embedded representation of a SELFIES molecule until it becomes pure noise.
  4. Reverse Process: Training a model to iteratively denoise the noisy representation, guided by the textual description, to reconstruct the original molecule embedding.
  5. Unified BioT5 Encoder: The model utilizes the encoder of the pre-trained BioT5 model. Crucially, this single encoder is used for both tokenized SELFIES strings and natural language text. This unified architecture simplifies the model and allows for efficient fine-tuning.
  6. Sampling Step Optimization: To speed up inference, the model employs a sampling strategy where it skips steps in the reverse diffusion process, predicting a less noisy vector from a more distant noisy one.

Datasets

The study uses the L+M-24 dataset, specifically the "split_train" for training and "split_valid" for evaluation. This dataset consists of paired SMILES strings and their corresponding descriptive captions.

  • Data Preprocessing: The training dataset was augmented with additional captions. Importantly, the SMILES strings were converted to SELFIES representations. A small number of molecules that could not be converted from SMILES to SELFIES were excluded. The evaluation dataset was kept in its original SMILES format for compatibility with existing metrics.

Key Insights and Ideas

  • SELFIES is Key to Validity: The paper strongly advocates for SELFIES as a superior representation for generative tasks due to its inherent validity guarantee, simplifying the generation process and eliminating the need for post-generation correction steps.
  • Unified Encoder Efficiency: Using a single BioT5 encoder for both text and SELFIES strings is a clever way to leverage pre-trained knowledge and streamline the model architecture, leading to better efficiency and potentially mitigating overfitting.
  • Diffusion for Molecule Generation is Viable: Extending diffusion models, which have excelled in image generation, to the domain of molecule generation shows significant promise, especially when combined with robust representations like SELFIES.

Key Results

  • State-of-the-Art Validity: Lang2Mol-Diff achieved a perfect 100.00% validity score on the L+M-24 benchmark dataset, outperforming all other evaluated methods.
  • Competitive Performance: While not outperforming all existing methods on every metric, the model achieved competitive scores across various metrics like BLEU, Levenshtein, and Morgan FTS.
  • Novelty: Qualitative comparisons suggest that Lang2Mol-Diff generates molecules with a higher degree of novelty compared to MolT5, especially when given distinct input descriptions.

Limitations and Future Work

  • Pre-training Dataset Size: The authors acknowledge that their model might not outperform existing methods on all metrics because it wasn't pre-trained on a dataset as large as some other models.
  • Model Configuration: The model was trained with default configurations similar to TGM-DLM, which might not be fully optimized for Lang2Mol-Diff.
  • Future Directions:
    • Larger Pre-training: Pre-training the model on a larger dataset could enhance its generalization capabilities.
  • Architecture and Hyperparameter Tuning: Exploring alternative model architectures and fine-tuning hyperparameters could further optimize performance.
  • Alternative Configurations: Investigating different diffusion schedules and sampling strategies could lead to further improvements.

Summary updated

BibTeX
@inproceedings{nguyen2024lang,
  author = {Nguyen Doan Hieu Nguyen and Nhat Truong Pham and Duong Thanh Tran and Balachandran Manavalan},
  title = {{Lang2Mol-Diff: A Diffusion-Based Generative Model for Language-to-Molecule Translation Leveraging SELFIES Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure