Token-Mol 1.0: Tokenized drug design with large language model

Jike Wang, Rui Qin, Mingyang Wang, Meijing Fang, Yangyang Zhang, Yuchen Zhu, Qun Su, Qiaolin Gou, Chao Shen, Odin Zhang, Zhenxing Wu, Dejun Jiang, Xujun Zhang, Huifeng Zhao, Xiaozhe Wan, Zhourui Wu, Liwei Liu, Yu Kang, Chang-Yu Hsieh, Tingjun HouarXiv 2024 · DOI · arXiv

Abstract

Token-Mol 1.0 is a unified, token-only large language model (LLM) designed to bridge the gap between 2D chemical sequences and 3D molecular structures. By representing torsion angles as discrete tokens and utilizing a novel Gaussian Cross-Entropy (GCE) loss, the model recasts drug design tasks—including conformation generation, property prediction, and pocket-based ligand design—as probabilistic prediction problems. Token-Mol 1.0 achieves state-of-the-art performance, notably operating up to 35 times faster than leading diffusion-based models while improving precision in 3D structural tasks.

Introduction

AI-driven drug discovery has traditionally faced a "language barrier" between sequence-based LLMs and geometry-focused Graph Neural Networks (GNNs). While SMILES-based LLMs excel at generating chemical strings, they often lack awareness of the 3D spatial arrangements (conformations) critical to biological function. Conversely, diffusion models and GNNs handle 3D geometry well but are difficult to integrate into flexible generative frameworks. Token-Mol 1.0 addresses this by enabling an LLM to "speak" 3D geometry through torsion angle tokenization, creating a cohesive architecture for diverse drug discovery tasks.

Methodology

Token-Mol 1.0 is built upon a 12-layer Transformer decoder architecture. Its core innovations involve data representation and loss functions tailored for chemical space.

Overview of Token-Mol 1.0 architecture, including pre-training/fine-tuning workflows and the GCE loss mechanism.

Torsion Angle Tokenization

Instead of raw Cartesian coordinates, the model utilizes torsion angles. This representation is compact, provides natural equivariance, and captures the essential degrees of freedom in a molecule without the overhead of 3D XYZ coordinates. This allows the model to handle 3D structures as efficiently as text.

Gaussian Cross-Entropy (GCE) Loss

To handle continuous values like binding affinity or physical properties, the authors introduced GCE loss. Unlike standard cross-entropy, which treats any incorrect token equally, GCE assigns weights based on a Gaussian distribution centered on the ground truth. This allows the model to learn the relative magnitude of numerical tokens more effectively, improving precision in regression tasks.

Training Strategy

The model employs Random Causal Masking to enhance "fill-in-the-blank" capabilities and uses the REINFORCE algorithm for fine-tuning. This allows the model to optimize for specific goals, such as drug-likeness (QED) and synthetic accessibility (SA), through reinforcement learning.

Key Results

Token-Mol 1.0 demonstrates significant improvements across multiple benchmarks:

  • Conformation Generation: Outperformed state-of-the-art (SOTA) methods by 10-20% in precision metrics across major datasets like GEOM-Drug.
  • Property Prediction: Achieved a 30% average improvement over existing token-only models (e.g., Regression Transformer) in regression tasks.
  • Pocket-based Design: Generated molecules showed an 11% increase in QED and a 14% improvement in SA compared to baselines, while maintaining high binding affinity.
  • Efficiency: The model achieved a 35x speedup over popular diffusion-based models for 3D generation, proving that autoregressive LLMs can be both accurate and highly efficient.

Conclusion

Token-Mol 1.0 proves that an autoregressive LLM can handle complex 3D molecular structures efficiently by treating numbers and geometric angles as tokens. While it currently lags slightly behind specialized GNNs in specific regression tasks, its unified paradigm and massive speed advantages make it a powerful tool for the future of drug design. Future work will explore larger datasets and advanced architectures like Mixture of Experts (MoE) and Retrieval-Augmented Generation (RAG).

Summary updated

Abstract

Significant interests have recently risen in leveraging sequence-based large language models (LLMs) for drug design. However, most current applications of LLMs in drug discovery lack the ability to comprehend three-dimensional (3D) structures, thereby limiting their effectiveness in tasks that explicitly involve molecular conformations. In this study, we introduced Token-Mol, a token-only 3D drug design model. This model encodes all molecular information, including 2D and 3D structures, as well as molecular property data, into tokens, which transforms classification and regression tasks in drug discovery into probabilistic prediction problems, thereby enabling learning through a unified paradigm. Token-Mol is built on the transformer decoder architecture and trained using random causal masking techniques. Additionally, we proposed the Gaussian cross-entropy (GCE) loss function to overcome the challenges in regression tasks, significantly enhancing the capacity of LLMs to learn continuous numerical values. Through a combination of fine-tuning and reinforcement learning (RL), Token-Mol achieves performance comparable to or surpassing existing task-specific methods across various downstream tasks, including pocket-based molecular generation, conformation generation, and molecular property prediction. Compared to existing molecular pre-trained models, Token-Mol exhibits superior proficiency in handling a wider range of downstream tasks essential for drug design. Notably, our approach improves regression task accuracy by approximately 30% compared to similar token-only methods. Token-Mol overcomes the precision limitations of token-only models and has the potential to integrate seamlessly with general models such as ChatGPT, paving the way for the development of a universal artificial intelligence drug design model that facilitates rapid and high-quality drug design by experts.
BibTeX
@misc{wang2024token,
  author = {Jike Wang and Rui Qin and Mingyang Wang and Meijing Fang and Yangyang Zhang and Yuchen Zhu and Qun Su and Qiaolin Gou and Chao Shen and Odin Zhang and Zhenxing Wu and Dejun Jiang and Xujun Zhang and Huifeng Zhao and Xiaozhe Wan and Zhourui Wu and Liwei Liu and Yu Kang and Chang-Yu Hsieh and Tingjun Hou},
  title = {{Token-Mol 1.0: Tokenized drug design with large language model}},
  year = {2024},
  eprint = {2407.07930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.07930},
}
Figure