TL;DR
This paper introduces an ensemble learning approach that combines features from multiple large protein language models and evolutionary coupling information to predict the effects of protein mutations. This method significantly improves prediction accuracy and generalization compared to existing state-of-the-art models, offering a more robust tool for protein engineering.
What Problem Does This Paper Address?
Predicting how mutations affect protein function is crucial for protein engineering, enabling the design of proteins with desired properties for applications in medicine, industry, and research. While large-scale protein language models (PLMs) have shown promise by learning evolutionary rules from vast sequence data, their full potential for quantitative prediction of mutation effects remains largely untapped. Existing methods often focus on building more complex individual models or utilize them in an unsupervised manner. This paper addresses the challenge of improving prediction accuracy and generalization by exploring how to effectively integrate features from multiple PLMs and other relevant biological information, and how to map these complex features to quantifiable functional changes.
Methods
The core of this work is an ensemble learning approach that integrates multiple sources of information.
- Global Features from Multiple PLMs: The researchers leverage pre-trained protein language models, specifically Tranception L (trained on UniRef100) and Tape-Transformer (trained on Pfam). Instead of fine-tuning, they use the "hidden states" from these models as feature extractors, generating "global features" that capture broad sequence characteristics.
- Local Features from Evolutionary Coupling: They incorporate local features derived from evolutionary coupling analysis using CCMPred. This algorithm identifies co-evolutionary relationships between amino acid pairs within homologous sequences, providing insights into residue interactions and their functional importance.
- Ensemble Representation: The global features from different PLMs are integrated (e.g., via summation after dimensionality reduction), and then concatenated with the local features to create a comprehensive "sequence representation."
- Supervised Prediction: This combined representation is then fed into a ridge regression model to predict quantifiable functional changes (mutation effects). The authors specifically chose ridge regression to investigate its effectiveness in mapping these complex features to function, comparing it implicitly to deep learning approaches.
Datasets
The primary dataset used for evaluation is S17, a curated subset of the ProteinGym dataset.
- ProteinGym: A comprehensive collection of deep mutational scanning (DMS) assay datasets covering diverse protein types and functional properties. It includes both substitution and indel mutation benchmarks.
- S17: This subset was specifically selected from the substitution benchmark to ensure:
- Inclusion of various taxa (humans, eukaryotes, prokaryotes, viruses).
- Even distribution of protein sequence lengths (100-1000 amino acids).
- A wide range of sample sizes (hundreds to thousands of mutants). This dataset comprises 17 distinct DMS datasets, aiming to provide a representative benchmark for evaluating model generalization across different protein types and experimental conditions.
Key Insights and Ideas
- Complementarity of PLMs: Different PLMs, trained on distinct datasets and with varying architectures, capture complementary information. Ensembling them is more powerful than using any single model alone.
- Local Features Enhance Global Features: Incorporating evolutionary coupling (local features) significantly enriches the information provided by PLMs (global features), leading to improved prediction performance. This suggests that a combination of broad evolutionary patterns and specific residue interaction information is key.
- Linear Models Can Be Powerful: The study demonstrates that even a relatively simple supervised model like ridge regression, when combined with rich ensemble features, can achieve state-of-the-art performance, challenging the notion that only complex deep learning architectures are sufficient for this task.
- Model Selectivity: Non-ensemble models (using only one PLM) can be selective, performing very well on some protein datasets but poorly on others. The ensemble approach mitigates this by providing a more consistent and robust performance across diverse datasets.
Key Results
- The proposed ensemble learning approach achieved an average Spearman rank correlation of 0.711 on the S17 dataset, representing a substantial improvement of approximately 70% over existing state-of-the-art supervised and unsupervised methods.
- The ensemble model outperformed non-ensemble methods on 15 out of 17 protein datasets in the S17 benchmark.
- The integration of local features (evolutionary coupling) improved the average Spearman rank correlation by approximately 0.1 across six different experimental scenarios.
- The ensemble method demonstrated superior performance even on small-scale datasets where individual models might struggle.
Limitations and Future Work
- Rare Undesired Performance: While generally superior, there were rare instances where the ensemble method did not outperform unsupervised models (e.g., on the A0A2Z5U3Z0 dataset). The authors attribute this to intricate relationships between measured attributes and protein fitness, or experimental inaccuracies.
- Focus on PLMs: The current approach primarily integrates protein language-based models.
- Future Directions: The authors suggest that future work should embrace multimodal deep learning approaches, integrating information beyond protein sequences (e.g., structural, functional, or experimental data). The proposed ensemble framework is seen as a crucial component for tackling such multimodal information. They also note that the integration approach can be extended to other pre-trained models beyond those used in this study.
Summary updated