Abstract
Token-Mol 1.0 is a unified, token-only large language model (LLM) designed to bridge the gap between 2D chemical sequences and 3D molecular structures. By representing torsion angles as discrete tokens and utilizing a novel Gaussian Cross-Entropy (GCE) loss, the model recasts drug design tasks—including conformation generation, property prediction, and pocket-based ligand design—as probabilistic prediction problems. Token-Mol 1.0 achieves state-of-the-art performance, notably operating up to 35 times faster than leading diffusion-based models while improving precision in 3D structural tasks.
Introduction
AI-driven drug discovery has traditionally faced a "language barrier" between sequence-based LLMs and geometry-focused Graph Neural Networks (GNNs). While SMILES-based LLMs excel at generating chemical strings, they often lack awareness of the 3D spatial arrangements (conformations) critical to biological function. Conversely, diffusion models and GNNs handle 3D geometry well but are difficult to integrate into flexible generative frameworks. Token-Mol 1.0 addresses this by enabling an LLM to "speak" 3D geometry through torsion angle tokenization, creating a cohesive architecture for diverse drug discovery tasks.
Methodology
Token-Mol 1.0 is built upon a 12-layer Transformer decoder architecture. Its core innovations involve data representation and loss functions tailored for chemical space.
Torsion Angle Tokenization
Instead of raw Cartesian coordinates, the model utilizes torsion angles. This representation is compact, provides natural equivariance, and captures the essential degrees of freedom in a molecule without the overhead of 3D XYZ coordinates. This allows the model to handle 3D structures as efficiently as text.
Gaussian Cross-Entropy (GCE) Loss
To handle continuous values like binding affinity or physical properties, the authors introduced GCE loss. Unlike standard cross-entropy, which treats any incorrect token equally, GCE assigns weights based on a Gaussian distribution centered on the ground truth. This allows the model to learn the relative magnitude of numerical tokens more effectively, improving precision in regression tasks.
Training Strategy
The model employs Random Causal Masking to enhance "fill-in-the-blank" capabilities and uses the REINFORCE algorithm for fine-tuning. This allows the model to optimize for specific goals, such as drug-likeness (QED) and synthetic accessibility (SA), through reinforcement learning.
Key Results
Token-Mol 1.0 demonstrates significant improvements across multiple benchmarks:
- Conformation Generation: Outperformed state-of-the-art (SOTA) methods by 10-20% in precision metrics across major datasets like GEOM-Drug.
- Property Prediction: Achieved a 30% average improvement over existing token-only models (e.g., Regression Transformer) in regression tasks.
- Pocket-based Design: Generated molecules showed an 11% increase in QED and a 14% improvement in SA compared to baselines, while maintaining high binding affinity.
- Efficiency: The model achieved a 35x speedup over popular diffusion-based models for 3D generation, proving that autoregressive LLMs can be both accurate and highly efficient.
Conclusion
Token-Mol 1.0 proves that an autoregressive LLM can handle complex 3D molecular structures efficiently by treating numbers and geometric angles as tokens. While it currently lags slightly behind specialized GNNs in specific regression tasks, its unified paradigm and massive speed advantages make it a powerful tool for the future of drug design. Future work will explore larger datasets and advanced architectures like Mixture of Experts (MoE) and Retrieval-Augmented Generation (RAG).
Summary updated