Multimodal Molecular Representation Learning

30 concepts21 papers1 proposal

To a computer, a molecule can be many things. It can be a linear string of characters (SMILES), a topological graph of atoms and bonds, a 2D rendering, a 3D geometric conformation, or a paragraph of scientific prose describing its behavior.

Historically, machine learning models in chemistry have been unimodal, relying on just one of these representations. However, every single modality operates under a fundamental trade-off: it highlights certain chemical properties while obscuring others. Linear strings are highly efficient for sequence-to-sequence modeling but obscure spatial proximity. Graphs capture connectivity perfectly but struggle with global geometric intuition or macro-level textual context.

To overcome these limitations, a major shift has emerged toward multimodal molecular representation learning. By fusing disparate views of the same chemical entity, these architectures allow models to cross-reference different structural and functional dimensions, building a holistic understanding of chemical space that no single modality can achieve alone.


The Intuition of Complementary Views

Why does combining modalities work better than simply scaling up a single representation? The core reason is that different formats contain non-redundant, complementary information about a molecule's underlying physical reality.

Consider the relationship between a 1D SMILES string and a 2D topological graph. As Xiangfeng Yan demonstrates in the development of Graph–sequence attention and transformer for predicting drug–target affinity, relying on a single modality inevitably discards critical structural or local context. A string representation forces a cyclic molecule into an arbitrary linear sequence, which can mask long-range topological relationships. A graph explicitly maps these connections but lacks the sequential grammar that allows Transformer architectures to easily model global context.

   SMILES String (Sequential Grammar)  <---\
                                            +---> Unified Molecular Representation
   Topological Graph (Local Connectivity) <--/

To exploit this complementarity, Ao Shen et al. introduced Complementary multi-modality molecular self-supervised learning via non-overlapping masking for property prediction. Their framework, MoleSG, uses a non-overlapping masking strategy during self-supervised pre-training. By masking parts of the SMILES string and different parts of the molecular graph simultaneously, the model is forced to use the visible graph structure to predict the missing string tokens, and vice versa. This forces the underlying network to learn the invariant chemical concepts that map between syntax and topology.


Aligning Modalities Across Scales

Fusing modalities requires careful architectural design. If you simply concatenate the final embeddings of a graph neural network and a language model, you miss the fine-grained interactions between individual atoms and their sequential counterparts.

To solve this, Bing Qin 0001 et al. proposed a multi-granularity approach in MolFusion: Multimodal Fusion Learning for Molecular Representations via Multi-granularity Views. Instead of fusing information only at the molecule level, their architecture aligns representations at both the molecular (global) and atomic (local) levels. This ensures that the specific token in a SMILES string representing a nitrogen atom is explicitly mapped to the corresponding node in the molecular graph, preserving local functional group nuances during downstream property prediction.

Global Level:   [ Molecule Embedding ] <=======> [ Graph Embedding ]
                       ^                                ^
                       |                                |
Local Level:    [ Atom Token (SMILES) ] <=====> [ Node Feature (Graph) ]

This multi-view alignment can be extended beyond 1D and 2D spaces. In MolMVC: Enhancing molecular representations for drug-related tasks through multi-view contrastive learning, Lei Deng et al. utilize multi-view contrastive learning to integrate 1D sequences, 2D structures, and 3D geometric conformations. Similarly, the authors of A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining developed MoleculeSDE, which uses group-symmetric diffusion models to maximize mutual information between 2D graphs and 3D geometric shapes directly in the input space, capturing intricate spatial details that are often lost in compressed latent spaces.


Integrating Visual and Geometric Intuition

Human chemists often reason about molecules by looking at diagrams. This visual intuition captures spatial layouts and symmetry in a way that symbolic strings cannot easily replicate. Recent work has successfully brought this visual modality into AI models by leveraging Vision-Language Models (VLMs).

In Reasoning-Enhanced Large Language Models for Molecular Property Prediction, Jiaxi Zhuang et al. introduced MPPReasoner, a multimodal LLM that combines visual diagrams of molecules with textual SMILES strings. This dual input allows the model to cross-reference the precise chemical connectivity of the string with the geometric layout of the image, significantly improving its ability to explain its chemical reasoning.

This approach is highly effective across specialized domains:


Bridging Structure and Natural Language

While structural modalities (graphs, strings, images) define what a molecule is, natural language describes what it does. Integrating scientific literature and textual descriptions allows models to capture macro-level biological behaviors and emergent properties.

As reviewed by Xiaoqi Wang et al. in Multimodal pre-training models of molecular representation for drug discovery, molecular captions act as a critical bridge to large language models. Early attempts at this integration were limited by sparse text data. To resolve this context bottleneck, Gang Liu et al. released MolTextNet: A Two-Million Molecule-Text Dataset for Multimodal Molecular Learning, which scales molecule-text pairs to 2.5 million samples. By expanding short labels into detailed scientific prose, they enabled text encoders to map structural features directly to complex biological behavior.

This synthesis of structure, text, and domain-specific features has unlocked powerful capabilities:


Resolving Ambiguity via Experimental Modalities

The ultimate frontier of multimodal molecular learning involves integrating raw experimental data, such as spectroscopy. Even when a molecule's theoretical structure is ambiguous or difficult to compute, its physical interaction with light or magnetic fields provides an unmistakable signature.

In a comprehensive survey of the field, Kehan Guo et al. outline in Artificial Intelligence in Spectroscopy: Advancing Chemistry from Prediction To Generation and Beyond how fusing multiple spectroscopic modalities (such as Mass Spectrometry, NMR, and IR) allows models to resolve structural ambiguities that single-modality approaches cannot handle.

To evaluate how well models can synthesize these disparate formats, Yuqing Huang et al. established ChemEval: A Multi-level and Fine-grained Chemical Capability Evaluation for Large Language Models. By testing models on tasks that require aligning molecular diagrams, spectral images, and text simultaneously, benchmarks like ChemEval highlight the core truth of modern chemical AI: true molecular understanding does not come from mastering a single format, but from learning to translate fluidly between them all.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multimodal_molecular_representation_8d6982c9,
  title = {multimodal molecular representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-molecular-representation/rdr_8d6982c9176241d8b01f56aeaa3eae85},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multimodal_molecular_representation_8d6982c9_v1,
  title = {multimodal molecular representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-molecular-representation/rdr_8d6982c9176241d8b01f56aeaa3eae85/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake