ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding

Yijia Xiao, Edward Sun, Yiqiao Jin, Qifan Wang, Wei Wang

TL;DR

ProteinGPT is a multimodal large language model (LLM) designed to bridge the gap between complex protein data and natural language. By integrating specialized sequence and structure encoders with a conversational LLM backbone, it allows researchers to upload protein files and ask detailed questions about their properties, functions, and structures in plain English.

What Problem Does This Paper Address?

Understanding a protein’s function and structure is the cornerstone of drug discovery and biotechnology, but it is traditionally a labor-intensive process involving manual literature searches and complex bioinformatics tools. While Large Language Models (LLMs) have revolutionized natural language processing, they struggle with proteins for two main reasons:

  1. Modality Gap: General-purpose LLMs cannot natively "read" 3D atomic coordinates or long, complex amino acid sequences.
  2. Unimodal Limitations: Existing protein-specific models often focus on either sequence or structure, missing the holistic view provided by combining both.

The authors of ProteinGPT aim to create a unified, multimodal system that can interpret diverse protein data types and provide accurate, contextually relevant answers to human inquiries.

Methods

The architecture of ProteinGPT follows a "frozen encoder + trainable bridge" strategy, similar to successful multimodal models in the vision-language space (like LLaVA).

  • The Encoders: The model uses two state-of-the-art, frozen pre-trained encoders: ESM-2 for protein sequences (capturing evolutionary and biochemical info) and ESM-IF1 for 3D structures (capturing spatial arrangements and dynamics).
  • The Projection Layer: To connect these encoders to the LLM, the authors introduced a linear projection layer. This layer acts as a translator, mapping protein embeddings into the LLM’s latent space.
  • The LLM Backbone: The researchers tested several open-source backbones, including Vicuna, Llama-2, Llama-3, and Mistral.

The training happens in two distinct stages:

  1. Modality Fusion & Alignment: The projection layer is trained to align protein representations with textual abstracts. In this stage, the model learns to generate a protein's summary based solely on its sequence and structure.
  2. Instruction Tuning: The entire system is fine-tuned on a massive question-answering (QA) dataset to ensure it can follow specific user instructions and provide concise answers.
ProteinGPT Modality Fusion & Alignment Stage

Datasets

The authors introduced ProteinQA, a large-scale dataset curated from the RCSB Protein Data Bank (PDB).

  • Scale: 132,092 protein samples.
  • Annotations: Each protein includes a descriptive abstract, 20–30 property tags, and 5–10 QA pairs.
  • Data Augmentation: To create high-quality instruction-tuning data, the authors used GPT-4o to decompose dense protein abstracts into granular, "atom-level" QA pairs. This ensures the model learns to answer specific questions rather than just reciting long summaries.
ProteinGPT Instruction Tuning Stage

Key Insights and Ideas

  • Multimodality is Essential: The paper demonstrates that combining sequence and structure data leads to significantly better understanding than using either alone. Sequences provide evolutionary context, while structures provide critical insights into binding sites and stability.
  • Mistral Wins on Sequences: Among the LLM backbones, Mistral performed the best. The authors attribute this to its Sliding Window Attention (SWA) mechanism, which is particularly effective at capturing dependencies in long amino acid chains.
  • General LLMs Struggle with Raw Data: Even powerful reasoning models like OpenAI’s o1-mini and DeepSeek-R1 performed poorly when fed raw protein sequences as text. This highlights the necessity of the specialized alignment and projection layers used in ProteinGPT.

Key Results

ProteinGPT significantly outperformed both vanilla open-source LLMs and proprietary general-purpose models on protein-related tasks.

  • Semantic Performance: ProteinGPT-Mistral achieved a BERTScore of 0.821, far exceeding the performance of vanilla Llama-3 (0.490) or even GPT-4o (0.596) when those models were given raw sequence data.
  • Factual Accuracy: In closed-ended accuracy tests (e.g., "How many assemblies does this protein have?"), the Mistral and Llama-3 variants achieved approximately 80% accuracy.
  • Efficiency: The model is optimized for speed, handling protein queries in roughly 50 seconds and generating responses in about 8 seconds on standard hardware.
Performance comparison across different stages and backbones

Limitations and Future Work

Despite its strong performance, the authors acknowledge several areas for improvement:

  • Hallucination: Like all LLMs, ProteinGPT can occasionally generate biologically inaccurate information. While grounding the model in structured data helps, it is not a perfect fix.
  • Verifiability: The current version does not provide direct citations for its claims, making it difficult for researchers to trace information back to primary literature.
  • Future Directions: The team plans to incorporate Retrieval-Augmented Generation (RAG) to ground answers in peer-reviewed literature and develop user-friendly interfaces for real-time collaboration in biological research.

Summary updated

Abstract

Understanding biological processes, drug development, and biotechnological advancements requires a detailed analysis of protein structures and functions, a task that is inherently complex and time-consuming in traditional protein research. To streamline this process, we introduce ProteinGPT, a state-of-the-art multimodal large language model for proteins that enables users to upload protein sequences and/or structures for comprehensive analysis and responsive inquiries. ProteinGPT integrates protein sequence and structure encoders with linear projection layers to ensure precise representation adaptation and leverages a large language model (LLM) to generate accurate, contextually relevant responses. To train ProteinGPT, we constructed a large-scale dataset of 132,092 proteins, each annotated with 20-30 property tags and 5-10 QA pairs per protein, and optimized the instruction-tuning process using GPT-4o. Experiments demonstrate that ProteinGPT effectively generates informative responses to protein-related questions, achieving high performance on both semantic and lexical metrics and significantly outperforming baseline models and general-purpose LLMs in understanding and responding to protein-related queries. Our code and data are available at https://github.com/ProteinGPT/ProteinGPT.
BibTeX
@misc{xiao2024proteingpt,
  author = {Yijia Xiao and Edward Sun and Yiqiao Jin and Qifan Wang and Wei Wang},
  title = {{ProteinGPT: Multimodal LLM for Protein Property Prediction and Structure Understanding}},
  year = {2024},
  eprint = {2408.11363},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.11363},
}
Figure