Protein model accuracy estimation empowered by deep learning and inter-residue distance prediction in CASP14

Xiao Chen, Jian Liu, Zhiye Guo, Tianqi Wu, Jie Hou, Jianlin Cheng

Abstract

The estimation of protein model accuracy (EMA) is a critical step in protein structure prediction. During the 2020 CASP14 experiment, the MULTICOM suite integrated inter-residue distance features with traditional quality assessment metrics using deep learning. The resulting multi-model predictors—MULTICOM-CONSTRUCT, MULTICOM-AI, and MULTICOM-CLUSTER—achieved the lowest GDT-TS score losses, ranking first, second, and third among all 68 predictors. These results highlight the value of spatial geometry and distance-based features in predicting structural quality.

Introduction: The Challenge of Model Selection

In structural biology, predicting a protein's 3D shape from its sequence often results in a large pool of candidate structures, or "decoys." The primary challenge is identifying which of these models is the most accurate without knowing the true experimental structure. This task, known as the Estimation of Model Accuracy (EMA), has historically relied on physical energy functions or binary contact maps. However, these methods often lack the precision needed to distinguish between high-quality models. This paper explores how granular spatial information, specifically inter-residue distances, can improve the ranking and scoring of these structural predictions.

Methodology: The MULTICOM Framework

The MULTICOM framework advances EMA by treating protein distance maps as 2D images and applying computer vision techniques to evaluate them. Instead of simple binary contacts, the system predicts full inter-residue distance maps using tools like DeepDist. These predicted maps are then compared against the actual distances in the 3D decoys using metrics such as the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR).

The MULTICOM framework integrates single-model and multi-model quality assessments with contact and distance predictions to estimate model accuracy.

The suite includes both "multi-model" predictors, which use consensus information from the entire pool of decoys, and "single-model" predictors, which evaluate a structure in isolation. This hybrid approach allows for robust ranking when a large pool is available and absolute scoring when it is not. The system utilizes a two-stage training strategy where initial predictor outputs are refined by a second deep learning model to produce the final quality score.

Key Insights: The Power of Distance Features

A significant finding of this work is that continuous distance predictions provide a much richer signal for deep learning than binary contact maps. By analyzing feature importance through SHAP (SHapley Additive exPlanations), the researchers confirmed that distance-based features are among the most influential inputs for the models.

SHAP analysis reveals that distance-based features and consensus scores (pairwiseScore) are the most important factors in predicting model quality.

The use of image-processing metrics like SSIM proved particularly effective. These metrics are designed to detect distortions in spatial geometry, making them ideal for identifying "blurry" or physically unrealistic protein models. The "wisdom of the crowd" remains a powerful signal, as evidenced by the high importance of the pairwise consensus score, but distance features provide the necessary granularity to refine these rankings.

Performance in CASP14

The MULTICOM suite demonstrated exceptional performance during the blind CASP14 competition:

  • Top Rankings: The three multi-model predictors swept the top three spots for GDT-TS loss out of 68 entries.
  • Quantitative Success: MULTICOM-CONSTRUCT achieved an average GDT-TS loss of 0.073, with a high Pearson correlation (~0.88) between predicted and actual quality scores.
  • Single-Model Efficacy: MULTICOM-DEEP ranked within the top 10 of single-model methods, proving that distance features enhance accuracy even without consensus data.

Conclusion and Future Directions

While CASP14 marked a significant success for MULTICOM, the emergence of highly accurate folding methods like AlphaFold2 presents new challenges. Future EMA models will need larger training datasets that include high-quality decoys from modern distance-based folding methods to better reflect the current state of the art. Additionally, moving toward end-to-end deep learning—where models learn features directly from 3D coordinates rather than relying on expert-curated metrics—represents the next frontier in protein quality assessment.

Summary updated

Abstract

The inter-residue contact prediction and deep learning showed the promise to improve the estimation of protein model accuracy (EMA) in the 13th Critical Assessment of Protein Structure Prediction (CASP13). To further leverage the improved inter-residue distance predictions to enhance EMA, during the 2020 CASP14 experiment, we integrated several new inter-residue distance features with the existing model quality assessment features in several deep learning methods to predict the quality of protein structural models. According to the evaluation of performance in selecting the best model from the models of CASP14 targets, our three multi-model predictors of estimating model accuracy (MULTICOM-CONSTRUCT, MULTICOM-AI, and MULTICOM-CLUSTER) achieve the averaged loss of 0.073, 0.079, and 0.081, respectively, in terms of the global distance test score (GDT-TS). The three methods are ranked first, second, and third out of all 68 CASP14 predictors. MULTICOM-DEEP, the single-model predictor of estimating model accuracy (EMA), is ranked within top 10 among all the single-model EMA methods according to GDT-TS score loss. The results demonstrate that inter-residue distance features are valuable inputs for deep learning to predict the quality of protein structural models. However, larger training datasets and better ways of leveraging inter-residue distance information are needed to fully explore its potentials.
BibTeX
@inproceedings{chen2021protein,
  author = {Xiao Chen and Jian Liu and Zhiye Guo and Tianqi Wu and Jie Hou and Jianlin Cheng},
  title = {{Protein model accuracy estimation empowered by deep learning and inter-residue distance prediction in CASP14}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-90303-6},
}
Figure