TL;DR
ProCyon is a new multimodal foundation model that uses protein sequence, structure, and natural language to predict and generate protein phenotypes. It achieves state-of-the-art performance on various tasks, including retrieving proteins based on complex descriptions and generating novel phenotype descriptions for poorly characterized proteins, paving the way for a more comprehensive understanding of the human proteome.
What Problem Does This Paper Address?
Despite significant advances, a substantial portion of human proteins (around 29%) still lack experimentally validated functions, and even well-annotated proteins often miss context-specific phenotypic insights. This makes it challenging to fully understand how living systems operate at the molecular level. Existing methods for predicting protein function often rely on homology or pre-defined categories, limiting their ability to capture the full spectrum of multiscale, context-dependent phenotypes and generalize to novel or poorly characterized proteins.
Methods
ProCyon is a multimodal foundation model built upon a Llama-3-8B language model, incorporating encoders for protein sequence (ESM-2 3B) and structure (GearNet). It processes inputs of natural language, protein sequences, and small molecular structures to achieve two main output modes: protein retrieval and phenotype generation. For retrieval, it projects protein sequences and natural language prompts into a unified latent space to identify matching proteins. For phenotype generation, it tokenizes multimodal inputs and uses the LLM for autoregressive text generation, allowing for open-ended descriptions beyond training data.
Datasets
The core training dataset is ProCyon-INSTRUCT, a novel instruction tuning dataset comprising over 33 million protein-phenotype instructions. This dataset spans five knowledge domains: function, disease, therapeutics, protein domains, and interaction. The data was curated from 12 sources and augmented using OpenAI's GPT models to increase phenotype diversity and include different levels of description.
Key Insights and Ideas
- Unifying Modalities with Natural Language: ProCyon's key innovation is using natural language as a unifying modality to bridge diverse knowledge domains and represent arbitrary protein phenotypes, moving beyond rigid ontologies.
- Beyond Homology: By integrating sequence, structure, and language, ProCyon can capture functional similarities that might be missed by homology-based methods, especially for proteins with divergent sequences but similar functions.
- Open-Ended Phenotype Generation: The model can generate novel, descriptive phenotypes for proteins, even those not well-characterized or whose functions were discovered after the model's training cutoff.
- Contextual Retrieval: ProCyon excels at retrieving proteins based on complex, multimodal prompts, including those involving pleiotropic functions, disease contexts, and drug mechanisms of action.
- Experimental Validation: The paper demonstrates strong experimental validation of ProCyon's predictions, particularly in identifying novel MS genes and elucidating disease mechanisms.
Key Results
- ProCyon-SPLIT (a version of ProCyon) achieved top performance in 5 out of 6 retrieval benchmarks and was among the top three in all tasks, outperforming homology-based and single-modal methods.
- It demonstrated strong retrieval performance even on rare or unseen phenotypes, with average Fmax values around 0.70-0.73 for many-, few-, and zero-shot splits.
- In question-answering tasks, ProCyon-SPLIT achieved an average accuracy of 74.9%, outperforming other models, including significantly larger LLMs when using protein sequences as input.
- For phenotype generation, ProCyon outperformed GPT-4o across various knowledge domains, as measured by BERTScore F1.
- ProCyon correctly identified drug-binding domains in proteins with high accuracy (68.6% of multi-domain proteins ranked the correct domain first).
- Experimental validation in multiple sclerosis identified novel candidate MS genes, with ZNF727, EVI2B, and MS4A7 showing significant MS enrichment.
Limitations and Future Work
- Conversational Capabilities: ProCyon's conversational abilities currently lag behind those of advanced LLMs, limiting multi-turn interactions for iterative hypothesis refinement. Future work could incorporate larger LLMs and more training data.
- Static Training Dataset: The model's training dataset is static, unable to leverage continually generated biological research. Integrating continual learning, retrieval-augmented generation, or agentic workflows is suggested.
- Data Modalities: Future datasets could expand protein representation by incorporating data from other species or including finer-grained annotations (e.g., tissue-specific, cell type-specific). Integration with other data modalities like transcriptomics and epigenomics is also proposed.
- Generative Protein Design: ProCyon's representations could be adapted for generative protein design, guiding diffusion models to create novel functional proteins.
Summary updated