TL;DR
This paper introduces UniMoT, a novel unified language model that treats molecules and text as equally important modalities. By developing a "molecule tokenizer" that converts molecular structures into discrete tokens, UniMoT enables large language models (LLMs) to process and generate both molecular and textual information in a unified, autoregressive manner, achieving state-of-the-art results on various molecular tasks.
What Problem Does This Paper Address?
The success of Large Language Models (LLMs) has spurred efforts to apply them to molecular science, enabling tasks like drug discovery and materials design. However, existing "molecular LLMs" often rely on adapter-based architectures. These methods struggle to truly integrate molecular and textual data, treating them unequally and lacking sufficient supervision for the molecular modality. This imbalance makes it difficult to achieve robust molecule-text alignment, particularly for tasks requiring text-to-molecule generation. The core challenge is to represent molecules in a way that LLMs can naturally process, similar to how they handle text, to enable seamless comprehension and generation across both domains.
Methods
UniMoT's key innovation is a tokenizer-based architecture that bridges the gap between molecular structures and LLM token representations.
- Molecule Tokenizer: This component transforms molecular structures into sequences of discrete "molecule tokens." It uses a Vector Quantization (VQ) approach, incorporating a Q-Former (similar to those used in vision-language models) to extract context-aware molecular features. Crucially, the Q-Former is designed with causal masks to ensure the generated queries have a left-to-right dependency, aligning with the unidirectional attention mechanisms of LLMs. These queries are then quantized into discrete tokens using a learnable codebook.
- Unified Representation: The generated molecule tokens are added to the LLM's vocabulary, creating a unified representation space for both molecules and text. This allows the LLM to treat molecule tokens as new "words."
- Autoregressive Training: UniMoT is trained using a standard LLM autoregressive objective, predicting the next token in a sequence. This enables both molecule-to-text (e.g., generating a description from a molecule) and text-to-molecule (e.g., generating a molecule from a description) tasks.
- Four-Stage Training: The model undergoes a structured training process:
- Causal Q-Former Pretraining: Focuses on learning to generate causal queries from molecular features.
- Molecule Tokenizer Pretraining: Optimizes the tokenizer to convert these queries into discrete tokens, with a focus on molecule reconstruction.
- Unified Molecule-Text Pretraining: Integrates the tokenizer with an LLM for joint autoregressive pretraining on molecule-text data.
- Task-Specific Instruction Tuning: Fine-tunes the model for specific downstream molecular tasks.
Datasets
The paper evaluates UniMoT on several standard molecular datasets:
- Molecule Comprehension: MoleculeNet benchmark for molecular property prediction (classification tasks: BBBP, Tox21, ToxCast, Sider, ClinTox, MUV, HIV, BACE).
- Molecule Captioning: PubChem and ChEBI-20 datasets for generating textual descriptions of molecules.
- Molecule Generation: Mol-Instructions benchmark for tasks like caption-guided molecule generation and reagent prediction. The USPTO dataset is also used for other generation tasks.
Key Insights and Ideas
- Treating Molecules as Language: The core idea is to represent molecules as a distinct "language" that LLMs can understand and generate. By discretizing molecular information into tokens, UniMoT achieves this by making molecules "speak" the same language as text.
- Causality is Key for LLMs: The use of a Causal Q-Former is critical. LLMs are inherently sequential and rely on causal dependencies. Ensuring molecule tokens also exhibit this property is crucial for effective integration and training.
- Tokenizer vs. Adapters: The paper argues that tokenizer-based approaches are superior to adapter-based methods for molecule-text tasks, especially generation, because they treat modalities more equally and avoid the limitations of directly generating SMILES strings.
- Unification for Generalization: By unifying representation and training across modalities, UniMoT becomes a versatile multi-modal generalist, capable of both understanding and creating molecular information.
Key Results
- State-of-the-Art Performance: UniMoT achieves state-of-the-art results across a wide range of molecule comprehension and generation tasks.
- Molecule Property Prediction: UniMoT shows consistent improvements over traditional graph learning methods and other molecular LLMs on eight classification datasets from MoleculeNet (e.g., achieving 92.89% ROC-AUC on MUV, compared to 91.48% for InstructMol).
- Molecule Captioning: UniMoT outperforms baselines like MolT5, MoMu, and InstructMol on the PubChem dataset, demonstrating superior molecule comprehension abilities. For example, it achieves a BLEU-2 score of 31.3, compared to 30.3 for 3D-MoLM.
- Molecule Generation: UniMoT generates valid molecules with higher structural similarity to targets in caption-guided generation and reagent prediction tasks, achieving an Exact Match score of 0.237 and BLEU score of 0.698 on caption-guided generation.
Limitations and Future Work
- Comprehension Sacrifice: While the unified token representation facilitates generation, the paper acknowledges that the quantization process for molecule tokens can lead to some information loss, potentially sacrificing a small amount of performance in pure comprehension tasks compared to continuous representations.
- Complex Tasks: UniMoT has not been extensively tested on more intricate tasks like molecule editing, which require precise structural modifications.
- Data Scarcity: The limited availability of annotated molecular data restricts UniMoT's ability to fully learn and generalize molecular structures and properties.
- Future Directions: The authors suggest addressing data scarcity as a crucial step. Expanding evaluations to a broader range of real-world scenarios is also recommended to better understand the model's robustness and generalizability.
Summary updated