Ensemble Learning with Supervised Methods Based on Large-Scale Protein Language Models for Protein Mutation Effects Prediction

Yang Qu, Zitong Niu, Qiaojiao Ding, Taowa Zhao, Tong Kong, Bing Bai, Jianwei Ma, Yitian Zhao, Jianping ZhengInternational Journal of Molecular Sciences 2023 · DOI

TL;DR

This paper introduces an ensemble learning approach that combines features from multiple large protein language models and evolutionary coupling information to predict the effects of protein mutations. This method significantly improves prediction accuracy and generalization compared to existing state-of-the-art models, offering a more robust tool for protein engineering.

What Problem Does This Paper Address?

Predicting how mutations affect protein function is crucial for protein engineering, enabling the design of proteins with desired properties for applications in medicine, industry, and research. While large-scale protein language models (PLMs) have shown promise by learning evolutionary rules from vast sequence data, their full potential for quantitative prediction of mutation effects remains largely untapped. Existing methods often focus on building more complex individual models or utilize them in an unsupervised manner. This paper addresses the challenge of improving prediction accuracy and generalization by exploring how to effectively integrate features from multiple PLMs and other relevant biological information, and how to map these complex features to quantifiable functional changes.

Methods

The core of this work is an ensemble learning approach that integrates multiple sources of information.

  • Global Features from Multiple PLMs: The researchers leverage pre-trained protein language models, specifically Tranception L (trained on UniRef100) and Tape-Transformer (trained on Pfam). Instead of fine-tuning, they use the "hidden states" from these models as feature extractors, generating "global features" that capture broad sequence characteristics.
  • Local Features from Evolutionary Coupling: They incorporate local features derived from evolutionary coupling analysis using CCMPred. This algorithm identifies co-evolutionary relationships between amino acid pairs within homologous sequences, providing insights into residue interactions and their functional importance.
  • Ensemble Representation: The global features from different PLMs are integrated (e.g., via summation after dimensionality reduction), and then concatenated with the local features to create a comprehensive "sequence representation."
  • Supervised Prediction: This combined representation is then fed into a ridge regression model to predict quantifiable functional changes (mutation effects). The authors specifically chose ridge regression to investigate its effectiveness in mapping these complex features to function, comparing it implicitly to deep learning approaches.

Datasets

The primary dataset used for evaluation is S17, a curated subset of the ProteinGym dataset.

  • ProteinGym: A comprehensive collection of deep mutational scanning (DMS) assay datasets covering diverse protein types and functional properties. It includes both substitution and indel mutation benchmarks.
  • S17: This subset was specifically selected from the substitution benchmark to ensure:
  • Inclusion of various taxa (humans, eukaryotes, prokaryotes, viruses).
  • Even distribution of protein sequence lengths (100-1000 amino acids).
  • A wide range of sample sizes (hundreds to thousands of mutants). This dataset comprises 17 distinct DMS datasets, aiming to provide a representative benchmark for evaluating model generalization across different protein types and experimental conditions.

Key Insights and Ideas

  • Complementarity of PLMs: Different PLMs, trained on distinct datasets and with varying architectures, capture complementary information. Ensembling them is more powerful than using any single model alone.
  • Local Features Enhance Global Features: Incorporating evolutionary coupling (local features) significantly enriches the information provided by PLMs (global features), leading to improved prediction performance. This suggests that a combination of broad evolutionary patterns and specific residue interaction information is key.
  • Linear Models Can Be Powerful: The study demonstrates that even a relatively simple supervised model like ridge regression, when combined with rich ensemble features, can achieve state-of-the-art performance, challenging the notion that only complex deep learning architectures are sufficient for this task.
  • Model Selectivity: Non-ensemble models (using only one PLM) can be selective, performing very well on some protein datasets but poorly on others. The ensemble approach mitigates this by providing a more consistent and robust performance across diverse datasets.

Key Results

  • The proposed ensemble learning approach achieved an average Spearman rank correlation of 0.711 on the S17 dataset, representing a substantial improvement of approximately 70% over existing state-of-the-art supervised and unsupervised methods.
  • The ensemble model outperformed non-ensemble methods on 15 out of 17 protein datasets in the S17 benchmark.
  • The integration of local features (evolutionary coupling) improved the average Spearman rank correlation by approximately 0.1 across six different experimental scenarios.
  • The ensemble method demonstrated superior performance even on small-scale datasets where individual models might struggle.

Limitations and Future Work

  • Rare Undesired Performance: While generally superior, there were rare instances where the ensemble method did not outperform unsupervised models (e.g., on the A0A2Z5U3Z0 dataset). The authors attribute this to intricate relationships between measured attributes and protein fitness, or experimental inaccuracies.
  • Focus on PLMs: The current approach primarily integrates protein language-based models.
  • Future Directions: The authors suggest that future work should embrace multimodal deep learning approaches, integrating information beyond protein sequences (e.g., structural, functional, or experimental data). The proposed ensemble framework is seen as a crucial component for tackling such multimodal information. They also note that the integration approach can be extended to other pre-trained models beyond those used in this study.

Summary updated

Abstract

Machine learning has been increasingly utilized in the field of protein engineering, and research directed at predicting the effects of protein mutations has attracted increasing attention. Among them, so far, the best results have been achieved by related methods based on protein language models, which are trained on a large number of unlabeled protein sequences to capture the generally hidden evolutionary rules in protein sequences, and are therefore able to predict their fitness from protein sequences. Although numerous similar models and methods have been successfully employed in practical protein engineering processes, the majority of the studies have been limited to how to construct more complex language models to capture richer protein sequence feature information and utilize this feature information for unsupervised protein fitness prediction. There remains considerable untapped potential in these developed models, such as whether the prediction performance can be further improved by integrating different models to further improve the accuracy of prediction. Furthermore, how to utilize large-scale models for prediction methods of mutational effects on quantifiable properties of proteins due to the nonlinear relationship between protein fitness and the quantification of specific functionalities has yet to be explored thoroughly. In this study, we propose an ensemble learning approach for predicting mutational effects of proteins integrating protein sequence features extracted from multiple large protein language models, as well as evolutionarily coupled features extracted in homologous sequences, while comparing the differences between linear regression and deep learning models in mapping these features to quantifiable functional changes. We tested our approach on a dataset of 17 protein deep mutation scans and indicated that the integrated approach together with linear regression enables the models to have higher prediction accuracy and generalization. Moreover, we further illustrated the reliability of the integrated approach by exploring the differences in the predictive performance of the models across species and protein sequence lengths, as well as by visualizing clustering of ensemble and non-ensemble features.
BibTeX
@article{qu2023ensemble,
  author = {Yang Qu and Zitong Niu and Qiaojiao Ding and Taowa Zhao and Tong Kong and Bing Bai and Jianwei Ma and Yitian Zhao and Jianping Zheng},
  title = {{Ensemble Learning with Supervised Methods Based on Large-Scale Protein Language Models for Protein Mutation Effects Prediction}},
  journal = {International Journal of Molecular Sciences},
  year = {2023},
  doi = {10.3390/ijms242216496},
}
Figure