Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment

Jia Song, Wanru Zhuang, Yujie Lin, Liang Zhang, Chunyan Li, Jinsong Su, Song He, Xiaochen Bo

TL;DR

Researchers have developed a new cross-modal retrieval model that significantly improves how we search for molecules using natural language. By introducing a memory-bank-based feature projector and second-order similarity losses, the model bridges the "modality gap" between text and molecular graphs, outperforming previous state-of-the-art results by 6.4% on the ChEBI-20 dataset.

What Problem Does This Paper Address?

In drug discovery, pharmacologists often need to find molecules with specific biological properties described in literature. While databases like PubChem are vast, many molecules lack expert-annotated text descriptions, making traditional search difficult. Cross-modal retrieval aims to solve this by learning a shared embedding space where a text description (e.g., "a molecule that inhibits protein X") and a molecular graph sit close together.

However, existing models face two major hurdles:

  1. The Modality Gap: Text (1D sequences) and molecules (2D/3D graphs) have fundamentally different topological structures. Simple linear layers are often insufficient to find common ground between them.
  2. First-Order Limitations: Most models rely on contrastive learning (e.g., "is this text-molecule pair a match?"). This is "first-order" similarity. It ignores "second-order" similarity—how the relationship between two molecules in the embedding space should mirror the relationship between their corresponding text descriptions.

Methods

The authors propose a novel architecture with two primary innovations:

1. Memory Bank Based Feature Projector

Instead of passing encoder outputs through a standard linear layer, the model uses a Memory Bank. This bank contains $n$ learnable "memory vectors" that act as semantic anchors. These vectors serve as queries in a cross-attention module, interacting with the outputs of the text encoder (SciBERT) and the molecule encoder (GCN). This forced interaction through a shared set of learnable anchors helps "bridge" the structural differences between the two modalities.

2. Second-Order Similarity Losses ($L_{u2u}$ and $L_{u2c}$)

The core idea here is that if Molecule A is similar to Molecule B, then Description A should be similarly related to Description B. The authors calculate four types of similarity distributions within a training batch:

  • Uni-modal: Text-to-Text and Molecule-to-Molecule.
  • Cross-modal: Text-to-Molecule and Molecule-to-Text.

They then use Kullback-Leibler (KL) divergence to minimize the distance between these distributions. Specifically, they use the highly reliable uni-modal distributions (from pre-trained encoders) as "soft labels" to guide and smooth the learning of the cross-modal alignments.

Datasets

The model was evaluated on two primary benchmarks:

  • ChEBI-20: A dataset of 33,010 compound-description pairs.
  • PCdes: A dataset used to test the model's generalizability and performance in "pretrain-finetune" paradigms.

Key Insights and Ideas

  • The "Bridge" Concept: The memory bank acts as a universal translator. By forcing both modalities to "talk" to the same set of memory vectors, the model learns features that are inherently modality-agnostic.
  • Relational Consistency: The shift from first-order to second-order similarity is crucial. It ensures that the structure of the embedding space is consistent. If the text embeddings form a cluster of "antibiotics," the molecular embeddings for those same items are pressured to form a similar cluster.
  • Leveraging Pre-training: The authors smartly recognize that uni-modal encoders (like SciBERT) are already very good at understanding relationships within their own domain. By using these internal relationships to supervise cross-modal learning, they provide a much richer training signal than simple binary (match/no-match) labels.

Key Results

The proposed model achieved new State-of-the-Art (SOTA) performance:

  • ChEBI-20 Retrieval: Achieved a Hits@1 of 56.5%, a significant jump over the previous best of 50.1% (Atomas).
  • Mean Rank Improvement: In text-to-molecule tasks, the Mean Rank dropped from 14.49 to 12.66, indicating a much more accurate sorting of potential candidates.
  • Ablation Success: Removing the memory bank led to a sharp decline in performance, proving it is a vital component for bridging the modality gap.
  • Reduced Modality Gap: Qualitative analysis (using Kernel Density Estimation) showed that the actual distance between text and molecule embeddings for the same instance was significantly smaller compared to previous models like AMAN.

Limitations and Future Work

While the model shows impressive gains, the authors note a few areas for growth:

  • LLM Integration: The current text encoder is SciBERT. The authors suggest that integrating Large Language Models (LLMs) could further enhance the nuances captured in text descriptions.
  • Knowledge Graphs: Future iterations may look into incorporating external knowledge bases to provide even more context during the retrieval process.
  • Scaling: While effective on ChEBI-20, testing the architecture on even larger, noisier datasets would be a logical next step for industrial application.

Summary updated

Abstract

Cross-modal text-molecule retrieval model aims to learn a shared feature space of the text and molecule modalities for accurate similarity calculation, which facilitates the rapid screening of molecules with specific properties and activities in drug design. However, previous works have two main defects. First, they are inadequate in capturing modality-shared features considering the significant gap between text sequences and molecule graphs. Second, they mainly rely on contrastive learning and adversarial training for cross-modality alignment, both of which mainly focus on the first-order similarity, ignoring the second-order similarity that can capture more structural information in the embedding space. To address these issues, we propose a novel cross-modal text-molecule retrieval model with two-fold improvements. Specifically, on the top of two modality-specific encoders, we stack a memory bank based feature projector that contain learnable memory vectors to extract modality-shared features better. More importantly, during the model training, we calculate four kinds of similarity distributions (text-to-text, text-to-molecule, molecule-to-molecule, and molecule-to-text similarity distributions) for each instance, and then minimize the distance between these similarity distributions (namely second-order similarity losses) to enhance cross-modal alignment. Experimental results and analysis strongly demonstrate the effectiveness of our model. Particularly, our model achieves SOTA performance, outperforming the previously-reported best result by 6.4%.
BibTeX
@inproceedings{song2024towards,
  author = {Jia Song and Wanru Zhuang and Yujie Lin and Liang Zhang and Chunyan Li and Jinsong Su and Song He and Xiaochen Bo},
  title = {{Towards Cross-Modal Text-Molecule Retrieval with Better Modality Alignment}},
  booktitle = {BIBM},
  year = {2024},
  url = {https://openreview.net/forum?id=1hiNH772vd},
}

Loading related content...

Figure