Omni-Mol: Exploring Universal Convergent Space for Omni-Molecular Tasks

Chengxin Hu, Hao Li, Yihe Yuan, Zezheng Song, Haixin Wang

TL;DR

Omni-Mol is a novel multimodal large language model (LLM) that unifies 16 diverse molecular tasks into a single framework, achieving state-of-the-art performance on 13 of them. By introducing a new large-scale dataset and innovative techniques like Gradient Adaptive LoRA (GAL) and Mixture-of-GAL-Experts (MoGE), Omni-Mol demonstrates the potential for a truly generalist AI chemist capable of both understanding and generating molecules.

What Problem Does This Paper Address?

The field of molecular AI has seen significant progress with multimodal LLMs, but a truly universal molecular model remains elusive. The authors identify three key challenges:

  1. Data Scarcity and Coverage: Existing molecular task datasets are often small and lack comprehensive domain coverage.
  2. Task Distributional Shifts: Different molecular subfields have distinct data distributions, making it hard for LLMs to learn tasks jointly without instability.
  3. Representation Dimensionality Mismatch: Inter- and intra-task molecular representations require different intrinsic dimensions in the language space, creating a challenge in balancing model capacity.

These issues hinder the development of a general-purpose AI expert for molecular tasks.

Methods

Omni-Mol tackles these challenges with several key innovations:

  • Task Categorization: Existing small-molecule tasks are categorized into four types: Mol2Mol, Mol2Text, Mol2Num, and Text2Mol.
  • Omni-Mol Dataset: A new, large-scale instruction-tuning dataset is created, encompassing over 16 tasks and more than 1.4 million samples, making it the largest of its kind.
  • Omni-Mol Framework: A novel multimodal LLM framework built on LLaMA 3, designed for unified molecular understanding and generation.
  • Gradient Adaptive LoRA (GAL): An adaptive mechanism extending LoRA that dynamically adjusts scaling factors to better handle varying intrinsic dimensions across tasks, mitigating learning conflicts.
  • Mixture-of-GAL-Experts (MoGE): A mixture-of-experts architecture that incorporates both specialized (routed) and general (shared) experts. This allows Omni-Mol to effectively capture common knowledge while differentiating across diverse tasks and modalities, enhancing its ability to handle inter-task and cross-modal interference.

Datasets

  • Omni-Mol Dataset: A newly constructed, comprehensive instruction-tuning dataset comprising over 16 tasks and more than 1.4 million samples. This dataset is designed to cover diverse molecular tasks and modalities.
  • Existing Datasets: The paper leverages and integrates data from various established sources, including USPTO, ChEBI-20, QM9, PubChem, RNX Yields, ORD, and ZINC.

Key Insights and Ideas

  • Unified Learning is Beneficial: Training a single model across a diverse set of molecular tasks (unified tuning) leads to better performance than training specialist models or using separate adapters for each task. This suggests that a generalist model can indeed learn a more cohesive and effective representation space.
  • Intrinsic Dimensionality Matters: Different molecular tasks have varying intrinsic dimensionalities, which GAL addresses by adaptively adjusting LoRA parameters. This is crucial for preventing conflicts and improving learning stability.
  • Mixture-of-Experts for Specialization: MoGE, with its combination of routed and shared experts, is key to handling the complexity of diverse molecular tasks. It allows the model to leverage both broad chemical knowledge and task-specific nuances.
  • Scalability is Key: Omni-Mol demonstrates strong scaling behavior with both increased data volume and model size, indicating its potential for further improvements with greater computational resources.
  • Convergent Representation Space: As Omni-Mol is trained on more tasks, its internal representations become increasingly similar, suggesting it's converging towards a universal space that captures fundamental molecular principles. This contrasts with specialist models that may diverge.

Key Results

  • Omni-Mol achieves state-of-the-art (SOTA) performance on 13 out of 16 tasks.
  • It outperforms specialist baselines by an average of approximately 5-40% across various tasks, while using only 33% of the parameters of some competitors.
  • On Mol2Num tasks, Omni-Mol improves R² score by 21.4% and lowers MAE by up to 39.3%.
  • The model shows significant improvements in molecule editing tasks, achieving higher QED scores and better physicochemical balance compared to baselines.
  • Experiments show that Omni-Mol scales effectively with dataset size and model size, with performance consistently improving.

Limitations and Future Work

  • Computational Resources: The authors acknowledge that limited computational resources prevented them from exploring the absolute limits of model performance by scaling up further.
  • Small Molecule Focus: The current tasks are primarily focused on small molecules. Future work should extend Omni-Mol to handle proteins and protein-small molecule interactions.

Summary updated

Abstract

In the molecular domain, numerous studies have explored the use of multimodal large language models (LLMs) to construct a general-purpose, multi-task molecular model. However, these efforts are still far from achieving a truly universal molecular model. We identify three key challenges in this endeavor: (1) Existing molecular task datasets are typically small in scale and lack comprehensive domain coverage. (2) Tasks from different molecular subfields are difficult to effectively learn jointly through LLMs due to significant distributional shifts and competition among tasks, which introduces instability in the learning process. (3) Both inter-task and intra-task molecular representations demand different intrinsic dimensions in the language space, making it challenging to balance between redundancy and insufficiency in language model representations. To address these challenges, we innovatively categorize existing small-molecule tasks into four types: Mol2Mol, Mol2Text, Mol2Num, and Text2Mol. We then collect a dataset encompassing over 16 tasks with more than 1.4 million samples, making it the largest molecular instruction-tuning dataset to date. Leveraging the extensive pretraining of LLMs on existing chemical literature, we propose a novel multimodal LLM framework, named Omni-Mol, which unifies all small-molecule tasks and supports both molecular generation and understanding. The core of Omni-Mol is our proposed MoGE, which dynamically adapts to the intrinsic rank of different tasks. This mixture-of-experts architecture enhances the model's ability to handle diverse tasks and modalities effectively. Our model achieves unified instruction tuning across 16 tasks and attains state-of-the-art performance on 13 of them. Extensive experiments further demonstrate the scalability and versatility of Omni-Mol.
BibTeX
@inproceedings{hu2025omni,
  author = {Chengxin Hu and Hao Li and Yihe Yuan and Zezheng Song and Haixin Wang},
  title = {{Omni-Mol: Exploring Universal Convergent Space for Omni-Molecular Tasks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=dw9H08UxJb},
}
Figure