Leveraging a large language model to predict protein phase transition: a physical, multiscale and interpretable approach

Mor Frank, Pengyu Ni, Matthew L. Jensen, Mark Gerstein

TL;DR

This paper introduces a novel approach using large language models (LLMs) to predict protein phase transitions (PPTs) – the shift from soluble proteins to liquid droplets or solid aggregates like amyloids. By fine-tuning an LLM, the researchers achieve superior prediction accuracy compared to traditional methods and demonstrate its utility in understanding how sequence mutations affect these transitions, offering new avenues for protein design and disease research.

What Problem Does This Paper Address?

Proteins, the workhorses of our cells, don't always stay in their simple, soluble form. They can undergo "phase transitions," transforming into either liquid-like droplets (crucial for cellular organization) or solid aggregates, such as the amyloids implicated in neurodegenerative diseases like Alzheimer's. While computational tools exist to predict these transitions, they often treat droplet formation and amyloid aggregation as separate problems, lacking a unified framework. Furthermore, understanding why a protein transitions and how specific sequence changes influence this behavior has been challenging, especially with the "black-box" nature of powerful AI models like LLMs. This paper aims to bridge this gap by developing a single, interpretable model that can predict both droplet and amyloid formation from protein sequences and analyze the impact of mutations.

Methods

The core of this work lies in leveraging and adapting a pre-trained large language model, specifically ESM-2, for the task of predicting protein phase transitions. The authors fine-tuned ESM-2, which is known for its efficiency in protein structure prediction, by adding a linear layer for classification. This approach allows the model to learn contextualized embeddings of protein sequences that capture information relevant to PPT.

To address the "black-box" nature of LLMs and enhance interpretability, the researchers also employed a classical machine learning approach: a Random Forest (RF) classifier. This RF model was trained on a set of carefully curated biophysical features derived from protein sequences, such as the fraction of disordered regions, gyration radius, instability index, solubility probability, and average hydrophobicity.

A key innovation is the creation of an ensemble model (ML_LM) that combines the predictions of both the fine-tuned LLM and the RF model, aiming to achieve the best of both worlds – the predictive power of the LLM and the interpretability of the biophysical features. The authors also developed a "transition score" to quantify a protein's propensity to undergo PPT, providing a continuous measure rather than a binary classification.

Datasets

The study utilized several datasets:

  • +Droplet drivers: Proteins experimentally shown to undergo liquid-liquid phase separation (LLPS) and form liquid or gel-like condensates. This dataset was constructed from three public databases: PhaSepDB, PhaSePro, and LLPSDB.
  • +Amyloids: Proteins with experimentally validated amyloidogenic regions, known to form solid aggregates. This dataset was derived from the AmyPro database.
  • -PT: A negative control dataset consisting of well-structured proteins randomly selected from the PDB, unlikely to undergo LLPS or form amyloid aggregates.

The authors specifically excluded proteins exhibiting both droplet and amyloid formation due to limited data, focusing on distinct behaviors.

Key Insights and Ideas

  • Unified PPT Prediction: The LLM approach successfully unifies the prediction of both droplet and amyloid formation within a single framework, a significant improvement over previous methods.
  • LLM Superiority: Fine-tuned LLMs, particularly when combined with biophysical features in an ensemble model, significantly outperform classical machine learning models in predicting PPT. This highlights the power of contextualized sequence embeddings.
  • Interpretable Insights: By comparing the LLM with an RF model trained on biophysical features, the researchers could identify key features driving PPT, such as the fraction of disordered regions and instability index.
  • Sequence Variants Matter: The model effectively predicts how sequence mutations impact PPT, opening doors for protein design and understanding disease-causing mutations.
  • AD Link: For Alzheimer's disease-related proteins (APP and tau), the study suggests a link between aggregation propensity and gene expression, hinting at a potential regulatory mechanism. Specifically, proteins with higher aggregation propensity (like APP) are downregulated in AD, potentially as a defense mechanism.

Key Results

  • High Predictive Accuracy: The LLM achieved high performance across various metrics (accuracy, recall, precision, AUROC), often exceeding 0.9 for classification tasks. The ensemble ML_LM model further improved these scores, particularly in recall.
  • Clearer Distinctions with LLM: Principal Component Analysis (PCA) showed that LLM embedding vectors provided a clearer distinction between protein classes (+Droplet drivers, +Amyloids, -PT) compared to biophysical features alone.
  • Feature Importance: For predicting PPT (task A), the "fraction of disordered regions" was a dominant feature. For distinguishing between droplets and amyloids (task B), the "instability index" and "fraction of disordered regions" were most important, suggesting thermodynamic drivers.
  • APP vs. Tau: The model predicted APP to have a higher propensity for amyloid aggregation, while tau was predicted to favor droplet formation, aligning with current hypotheses about their roles in Alzheimer's disease progression.
  • Gene Expression Correlation: A significant negative correlation was observed between the transition score (indicating aggregation propensity) and gene expression levels for APP and tau in AD samples, suggesting a regulatory feedback loop.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the limited availability of data, particularly for proteins exhibiting both droplet and amyloid formation, influenced their analysis.
  • Interpretability Trade-off: While the ensemble model improves interpretability, the inherent "black-box" nature of LLMs still presents challenges in fully understanding all decision-making processes.
  • Experimental Validation: The study relies on existing experimental data. Further experimental validation of the model's predictions, especially regarding sequence variants and gene expression links, is crucial.
  • Time-Dependent Dynamics: The current model predicts propensity but doesn't fully capture the dynamic, time-dependent nature of phase transitions, which could be a future research direction.
  • Broader Applications: The authors suggest that this framework could be extended to design antibodies with reduced aggregation propensity and to explore other age-related diseases. The potential for leveraging droplet formation as a therapeutic target is also highlighted.

Summary updated

Abstract

Abstract Protein phase transitions (PPTs) from the soluble state to a dense liquid phase (forming droplets via liquid-liquid phase separation) or to solid aggregates (such as amyloids) play key roles in pathological processes associated with age-related diseases such as Alzheimer’s disease. Several computational frameworks are capable of separately predicting the formation of droplets or amyloid aggregates based on protein sequences, yet none have tackled the prediction of both within a unified framework. Recently, large language models (LLMs) have exhibited great success in protein structure prediction; however, they have not yet been used for PPTs. Here, we fine-tune a LLM for predicting PPTs and demonstrate its usage in evaluating how sequence variants affect PPTs, an operation useful for protein design. In addition, we show its superior performance compared to suitable classical benchmarks. Due to the ”black-box” nature of the LLM, we also employ a classical random forest model along with biophysical features to facilitate interpretation. Finally, focusing on Alzheimer’s disease-related proteins, we demonstrate that greater aggregation is associated with reduced gene expression in AD, suggesting a natural defense mechanism. Significance Statement Protein phase transition (PPT) is a physical mechanism associated with both physiological processes and age-related diseases. We present a modeling approach for predicting the protein propensity to undergo PPT, forming droplets or amyloids, directly from its sequence. We utilize a large language model (LLM) and demonstrate how variants within the protein sequence affect PPT. Because the LLM is naturally domain-agnostic, to enhance interpretability, we compare it with a classical knowledge-based model. Furthermore, our findings suggest the possible regulation of PPT by gene expression and transcription factors, hinting at potential targets for drug development. Our approach demonstrates the usefulness of fine-tuning a LLM for downstream tasks where only small datasets are available.
BibTeX
@inproceedings{frank2024leveraging,
  author = {Mor Frank and Pengyu Ni and Matthew L. Jensen and Mark Gerstein},
  title = {{Leveraging a large language model to predict protein phase transition: a physical, multiscale and interpretable approach}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2024},
  doi = {10.1073/pnas.2320510121},
}
Figure