ProCyon: A multimodal foundation model for protein phenotypes

Owen Queen, Yepeng Huang, Robert Calef, Valentina Giunchiglia, Tianlong Chen, George Dasoulas, L. Tai, Yasha Ektefaie, Ayush Noori, Joseph S. Brown, Tom Cobley, Karin Hrovatin, Thomas Hartvigsen, Fabian J. Theis, Bradley L. Pentelute, Vikram Khurana, Manolis Kellis, Marinka Žitnik, James Zou, David Owen, Richard NicholasbioRxiv 2024 · DOI

TL;DR

ProCyon is a new multimodal foundation model that uses protein sequence, structure, and natural language to predict and generate protein phenotypes. It achieves state-of-the-art performance on various tasks, including retrieving proteins based on complex descriptions and generating novel phenotype descriptions for poorly characterized proteins, paving the way for a more comprehensive understanding of the human proteome.

What Problem Does This Paper Address?

Despite significant advances, a substantial portion of human proteins (around 29%) still lack experimentally validated functions, and even well-annotated proteins often miss context-specific phenotypic insights. This makes it challenging to fully understand how living systems operate at the molecular level. Existing methods for predicting protein function often rely on homology or pre-defined categories, limiting their ability to capture the full spectrum of multiscale, context-dependent phenotypes and generalize to novel or poorly characterized proteins.

Methods

ProCyon is a multimodal foundation model built upon a Llama-3-8B language model, incorporating encoders for protein sequence (ESM-2 3B) and structure (GearNet). It processes inputs of natural language, protein sequences, and small molecular structures to achieve two main output modes: protein retrieval and phenotype generation. For retrieval, it projects protein sequences and natural language prompts into a unified latent space to identify matching proteins. For phenotype generation, it tokenizes multimodal inputs and uses the LLM for autoregressive text generation, allowing for open-ended descriptions beyond training data.

Datasets

The core training dataset is ProCyon-INSTRUCT, a novel instruction tuning dataset comprising over 33 million protein-phenotype instructions. This dataset spans five knowledge domains: function, disease, therapeutics, protein domains, and interaction. The data was curated from 12 sources and augmented using OpenAI's GPT models to increase phenotype diversity and include different levels of description.

Key Insights and Ideas

  • Unifying Modalities with Natural Language: ProCyon's key innovation is using natural language as a unifying modality to bridge diverse knowledge domains and represent arbitrary protein phenotypes, moving beyond rigid ontologies.
  • Beyond Homology: By integrating sequence, structure, and language, ProCyon can capture functional similarities that might be missed by homology-based methods, especially for proteins with divergent sequences but similar functions.
  • Open-Ended Phenotype Generation: The model can generate novel, descriptive phenotypes for proteins, even those not well-characterized or whose functions were discovered after the model's training cutoff.
  • Contextual Retrieval: ProCyon excels at retrieving proteins based on complex, multimodal prompts, including those involving pleiotropic functions, disease contexts, and drug mechanisms of action.
  • Experimental Validation: The paper demonstrates strong experimental validation of ProCyon's predictions, particularly in identifying novel MS genes and elucidating disease mechanisms.

Key Results

  • ProCyon-SPLIT (a version of ProCyon) achieved top performance in 5 out of 6 retrieval benchmarks and was among the top three in all tasks, outperforming homology-based and single-modal methods.
  • It demonstrated strong retrieval performance even on rare or unseen phenotypes, with average Fmax values around 0.70-0.73 for many-, few-, and zero-shot splits.
  • In question-answering tasks, ProCyon-SPLIT achieved an average accuracy of 74.9%, outperforming other models, including significantly larger LLMs when using protein sequences as input.
  • For phenotype generation, ProCyon outperformed GPT-4o across various knowledge domains, as measured by BERTScore F1.
  • ProCyon correctly identified drug-binding domains in proteins with high accuracy (68.6% of multi-domain proteins ranked the correct domain first).
  • Experimental validation in multiple sclerosis identified novel candidate MS genes, with ZNF727, EVI2B, and MS4A7 showing significant MS enrichment.

Limitations and Future Work

  • Conversational Capabilities: ProCyon's conversational abilities currently lag behind those of advanced LLMs, limiting multi-turn interactions for iterative hypothesis refinement. Future work could incorporate larger LLMs and more training data.
  • Static Training Dataset: The model's training dataset is static, unable to leverage continually generated biological research. Integrating continual learning, retrieval-augmented generation, or agentic workflows is suggested.
  • Data Modalities: Future datasets could expand protein representation by incorporating data from other species or including finer-grained annotations (e.g., tissue-specific, cell type-specific). Integration with other data modalities like transcriptomics and epigenomics is also proposed.
  • Generative Protein Design: ProCyon's representations could be adapted for generative protein design, guiding diffusion models to create novel functional proteins.

Summary updated

Abstract

Characterizing human proteins remains a major challenge: approximately 29% of human proteins lack experimentally validated functions and even well-annotated proteins often lack context-specific phenotypic insights. To enable universal modeling of protein phenotypes, we present ProCyon, a multimodal foundation model that utilizes protein sequence, structure, and natural language for generating and predicting protein phenotypes across diverse knowledge domains. ProCyon is trained on our novel dataset, ProCyon-Instruct, with 33 million protein phenotype instructions. On dozens of benchmarking tasks, ProCyon performs competitively against single-modal and multimodal models. Further, ProCyon conditionally retrieves proteins via mechanisms of action of small molecule drugs and disease contexts, and it generates candidate phenotypic descriptions for poorly characterized proteins, including those implicated in Parkinson's disease that were identified after ProCyon's knowledge cutoff date. We experimentally confirm ProCyon's predictions in multiple sclerosis using post-mortem brain RNA-seq, identifying novel MS genes and elucidating associated pathway mechanisms consistent with cortical pathology. ProCyon paves the way toward a general approach to generate functional insights into the human proteome.
BibTeX
@misc{queen2024procyon,
  author = {Owen Queen and Yepeng Huang and Robert Calef and Valentina Giunchiglia and Tianlong Chen and George Dasoulas and L. Tai and Yasha Ektefaie and Ayush Noori and Joseph S. Brown and Tom Cobley and Karin Hrovatin and Thomas Hartvigsen and Fabian J. Theis and Bradley L. Pentelute and Vikram Khurana and Manolis Kellis and Marinka Žitnik and Bradley L. Pentelute and James Zou and Vikram Khurana and David Owen and Richard Nicholas and Manolis Kellis and Marinka Zitnik},
  title = {{ProCyon: A multimodal foundation model for protein phenotypes}},
  year = {2024},
  doi = {10.1101/2024.12.10.627665},
  howpublished = {bioRxiv},
}
Figure