Empirical comparison and analysis of machine learning-based predictors for predicting and analyzing of thermophilic proteins

Phasit Charoenkwan, Nalini Schaduangrat, Md Mehedi Hasan, Mohammad Ali Moni, Píetro Lió, Watshara ShoombuatongPubMed 2022 · DOI

TL;DR

This paper provides the first comprehensive review and empirical comparison of machine learning (ML)-based predictors for identifying thermophilic proteins (TPPs). By evaluating fourteen state-of-the-art methods across multiple datasets and criteria, the authors highlight the strengths and weaknesses of existing approaches, offering valuable insights for researchers and guiding future development of more accurate TPP prediction tools.

What Problem Does This Paper Address?

Thermophilic proteins (TPPs) are crucial for various applications, from basic research to industrial processes, due to their remarkable stability at high temperatures. Identifying these proteins efficiently is essential, but experimental methods are time-consuming and costly. While computational methods, particularly those using machine learning (ML), have emerged for in silico prediction of TPPs, there's a lack of a consolidated overview and comparative analysis of these diverse tools. This paper aims to fill that gap by systematically reviewing and comparing existing ML-based TPP predictors, assessing their methodologies, performance, and usability to guide researchers and inform future advancements.

Methods

The authors conducted a comprehensive review and empirical comparison of fourteen existing sequence-based TPP predictors. Their analysis focused on several key aspects:

  • Dataset Preparation: Examining the training and independent datasets used for developing these predictors.
  • Feature Extraction and Optimization: Investigating the various methods used to represent protein sequences as numerical features and strategies for selecting the most informative ones.
  • ML Algorithms: Categorizing predictors based on the ML algorithms employed, distinguishing between "black-box" (less interpretable) and "white-box" (more interpretable) models.
  • Evaluation Strategies: Analyzing how the performance of these predictors was assessed (e.g., cross-validation, independent testing).
  • Web Server/Software Usability: Evaluating the accessibility and practical utility of the predictors for researchers.

The comparative analysis involved evaluating several predictors on two benchmark datasets.

Datasets

The paper utilized and discussed several datasets for training and evaluating TPP predictors:

  • Zhang2006: A dataset used for training and as an independent test set.
  • Zhang2007: Another dataset used for training and as an independent test set.
  • Gromiha2007: A well-known training dataset, derived from Zhang2007 with sequence redundancy reduced.
  • Lin2011: Another widely used training dataset, containing TPPs and non-TPPs extracted from UniProt. This dataset is available via a web server.
  • Charoenkwan2021: A newly constructed, up-to-date dataset comprising 1823 TPPs and 3124 non-TPPs, created by combining data from previous studies and reducing redundancy. This dataset is also available for download.

The paper also refers to various other datasets used by individual predictors, detailed in Table 2.

Key Insights and Ideas

  • Interpretability Matters: The study categorizes predictors into "black-box" and "white-box" models, highlighting that while black-box models (like SVM) often achieve high accuracy, white-box models (like SCM) offer better interpretability, which can be crucial for understanding the underlying biological mechanisms.
  • No Single "Best" Predictor: The "best" predictor depends on the specific needs. ThermoPred excels in predictive performance and community utility (due to its web server), while SCMTPP stands out for its high interpretability and simplicity.
  • Feature Engineering is Key: The choice of feature encoding schemes (e.g., amino acid composition, dipeptide composition, physicochemical properties) significantly impacts predictor performance.
  • Web Servers Enhance Utility: Predictors implemented as accessible web servers (like ThermoPred and SCMTPP) have much greater practical utility for experimental scientists compared to those that are not.
  • Limitations of Current Approaches: Existing predictors primarily rely on sequence-based features and single ML algorithms, suggesting room for improvement through more advanced techniques.

Key Results

  • Performance Comparison:
    • On the Gromiha2007 dataset, PSSM400_pKa achieved the highest ACC (93.53%), Sn (89.50%), and Sp (95.64%). ThermoPred showed the second-best performance.
  • On the Lin2011 dataset, Feng et al.'s method achieved the best performance with ACC, Sn, and Sp all at 98.20%. GA-MLR followed with strong results.
  • ThermoPred consistently demonstrated competitive performance across datasets and was noted for its balance of predictive accuracy and community utility.
  • SCMTPP showed comparable performance to ThermoPred on the Charoenkwan2021 dataset and was highlighted for its interpretability.
  • Model Complexity: ThermoPred (10 features) was found to be more computationally efficient than PSSM400_pKa (460 features).
  • Web Server Availability: Only ThermoPred and SCMTPP were implemented as web servers among the fourteen predictors reviewed, significantly enhancing their practical applicability.

Limitations and Future Work

  • Limitations:
    • Many predictors lack web server implementations, limiting their accessibility and utility for experimental researchers.
  • The reliance on sequence-based features might be a bottleneck for achieving higher accuracy.
  • The use of single ML algorithms can lead to models that are not robust in all cases.
  • Future Work: The authors propose several promising directions:
  • Sequence-to-Vector Encodings: Developing new models trained using advanced sequence-to-vector encodings, which have shown superior performance in other biological sequence analysis tasks.
  • Deep Learning (DL) Models: Exploring DL-based methods, which can automatically extract features from sequences without manual encoding, potentially leading to improved performance.
  • Ensemble Learning Strategies: Implementing ensemble methods (e.g., stacking) to combine multiple models and improve robustness and predictive accuracy.
  • Web Server Development: Encouraging the development of more web servers for existing and new TPP predictors to enhance their practical impact.

Summary updated

Abstract

Thermophilic proteins (TPPs) are critical for basic research and in the food industry due to their ability to maintain a thermodynamically stable fold at extremely high temperatures. Thus, the expeditious identification of novel TPPs through computational models from protein sequences is very desirable. Over the last few decades, a number of computational methods, especially machine learning (ML)-based methods, for <i>in silico</i> prediction of TPPs have been developed. Therefore, it is desirable to revisit these methods and summarize their advantages and disadvantages in order to further develop new computational approaches to achieve more accurate and improved prediction of TPPs. With this goal in mind, we comprehensively investigate a large collection of fourteen state-of-the-art TPP predictors in terms of their dataset size, feature encoding schemes, feature selection strategies, ML algorithms, evaluation strategies and web server/software usability. To the best of our knowledge, this article represents the first comprehensive review on the development of ML-based methods for in <i>silico</i> prediction of TPPs. Among these TPP predictors, they can be classified into two groups according to the interpretability of ML algorithms employed (i.e., computational black-box methods and computational white-box methods). In order to perform the comparative analysis, we conducted a comparative study on several currently available TPP predictors based on two benchmark datasets. Finally, we provide future perspectives for the design and development of new computational models for TPP prediction. We hope that this comprehensive review will facilitate researchers in selecting an appropriate TPP predictor that is the most suitable one to deal with their purposes and provide useful perspectives for the development of more effective and accurate TPP predictors.
BibTeX
@inproceedings{charoenkwan2022empirical,
  author = {Phasit Charoenkwan and Nalini Schaduangrat and Md Mehedi Hasan and Mohammad Ali Moni and Píetro Lió and Watshara Shoombuatong},
  title = {{Empirical comparison and analysis of machine learning-based predictors for predicting and analyzing of thermophilic proteins}},
  booktitle = {PubMed},
  year = {2022},
  doi = {10.17179/excli2022-4723},
}
Figure