Exploring the Alignment Landscape: LLMs and Geometric Deep Models in Protein Representation

Dong Shu, Bingbing Duan, Kai Guo, Kaixiong Zhou, Jiliang Tang, Mengnan Du

Introduction

Proteins are the fundamental machinery of life, defined not just by their linear amino acid sequences but by their intricate three-dimensional folds. While Large Language Models (LLMs) have demonstrated remarkable reasoning capabilities over textual data, they often lack an inherent understanding of physical geometry. Conversely, Geometric Deep Models (GDMs) are designed to process 3D coordinates but lack the communicative breadth of LLMs. This research addresses the challenge of latent representation alignment: synchronizing the abstract vector spaces of LLMs and GDMs to enable more accurate, hallucination-free multimodal protein analysis.

Methodology: The Alignment Framework

The authors propose a systematic framework to bridge the gap between textual and geometric modalities. The approach involves three primary stages: data preprocessing, representation extraction, and representation alignment.

The multimodal alignment framework, illustrating data preprocessing, representation extraction via LLMs and GDMs, and the alignment process using projection heads and contrastive learning.

Model Selection and Architecture

The study evaluated a "triplet" of components:

  1. LLMs: Gemma2-2B, Llama3.1-8B, and Llama3.1-70B were used to process textual descriptions.
  2. GDMs: Four specialized models—GearNet, GVP, ScanNet, and GAT—were selected to extract structural features.
  3. Projection Heads: To map GDM embeddings into the LLM's latent space, the researchers tested various architectures, identifying a two-layer MLP with ReLU activation as the optimal "bridge."

Training and Data Augmentation

Alignment was achieved through contrastive learning using a modified InfoNCE loss, which encourages the model to cluster corresponding text and structure representations while separating dissimilar ones. To improve the quality of textual targets, GPT-4o was employed to transform raw FASTA metadata into descriptive natural language. The primary dataset was derived from the RCSB Protein Data Bank (PDB), with additional evaluation on the Enzyme Commission (EC) dataset.

Results and Discussion

The experimental results highlight several critical factors in successful multimodal alignment.

Alignment performance across different LLM-GDM pairings (A), correlation between model representations (B), and the impact of embedding dimensions on alignment scores (C).

The Role of Geometry and Scale

A key finding is that structural richness matters. Models incorporating 3D coordinates (ScanNet and GearNet) consistently outperformed those relying on 2D graph relationships (GAT). Furthermore, larger LLMs demonstrated a superior capacity for alignment, likely due to their higher-dimensional embedding spaces providing more "room" to integrate complex geometric information. As shown in Figure 3, the combination of ScanNet and Llama3.1-70B yielded the highest alignment scores.

Addressing Rarity Bias

The research identified a significant "rarity bias" in protein datasets. "Popular" proteins—those extensively documented in literature—align much more effectively than "rare" proteins. This imbalance contributes to model hallucinations, where the AI may generate plausible but incorrect functional descriptions for less common structures. Implementing Retrieval-Augmented Generation (RAG) helped mitigate this by leveraging similar proteins in the latent space to guide the model.

Impact on Hallucinations

Better-aligned models showed a marked reduction in scientific hallucinations. For instance, while poorly aligned models might generically label a protein as a "helicase," well-aligned versions correctly identified specific functions, such as "spectinomycin phosphotransferase," based on structural cues.

Conclusion and Future Directions

This work establishes that effective alignment between LLMs and GDMs is essential for reliable protein-focused AI. While projection complexity and model scale are important, addressing data imbalances and homology—where structurally similar proteins have distinct IDs—remains a challenge. Future research should explore more sophisticated fusion mechanisms and the development of balanced datasets to ensure models are equally "literate" across the entire protein landscape.

Summary updated

Abstract

Latent representation alignment has become a foundational technique for constructing multimodal large language models (MLLM) by mapping embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs) to enable effective cross-modal understanding. While preliminary protein-focused MLLMs have emerged, they have predominantly relied on heuristic approaches, lacking a fundamental understanding of optimal alignment practices across representations. In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain. We comprehensively evaluate three state-of-the-art LLMs (Gemma2-2B, LLaMa3.1-8B, and LLaMa3.1-70B) with four protein-specialized GDMs (GearNet, GVP, ScanNet, GAT). Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process. Our key findings reveal that GDMs incorporating both graph and 3D structural information align better with LLMs, larger LLMs demonstrate improved alignment capabilities, and protein rarity significantly impacts alignment performance. We also find that increasing GDM embedding dimensions, using two-layer projection heads, and fine-tuning LLMs on protein-specific data substantially enhance alignment quality. These strategies offer potential enhancements to the performance of protein-related multimodal models. Our code and data are available at https://github.com/Tizzzzy/LLM-GDM-alignment.
BibTeX
@article{shu2025exploring,
  author = {Dong Shu and Bingbing Duan and Kai Guo and Kaixiong Zhou and Jiliang Tang and Mengnan Du},
  title = {{Exploring the Alignment Landscape: LLMs and Geometric Deep Models in Protein Representation}},
  journal = {Patterns},
  year = {2025},
  doi = {10.1016/j.patter.2025.101227},
}
Figure