Abstract
The estimation of protein model accuracy (EMA) is a critical step in protein structure prediction. During the 2020 CASP14 experiment, the MULTICOM suite integrated inter-residue distance features with traditional quality assessment metrics using deep learning. The resulting multi-model predictors—MULTICOM-CONSTRUCT, MULTICOM-AI, and MULTICOM-CLUSTER—achieved the lowest GDT-TS score losses, ranking first, second, and third among all 68 predictors. These results highlight the value of spatial geometry and distance-based features in predicting structural quality.
Introduction: The Challenge of Model Selection
In structural biology, predicting a protein's 3D shape from its sequence often results in a large pool of candidate structures, or "decoys." The primary challenge is identifying which of these models is the most accurate without knowing the true experimental structure. This task, known as the Estimation of Model Accuracy (EMA), has historically relied on physical energy functions or binary contact maps. However, these methods often lack the precision needed to distinguish between high-quality models. This paper explores how granular spatial information, specifically inter-residue distances, can improve the ranking and scoring of these structural predictions.
Methodology: The MULTICOM Framework
The MULTICOM framework advances EMA by treating protein distance maps as 2D images and applying computer vision techniques to evaluate them. Instead of simple binary contacts, the system predicts full inter-residue distance maps using tools like DeepDist. These predicted maps are then compared against the actual distances in the 3D decoys using metrics such as the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR).
The suite includes both "multi-model" predictors, which use consensus information from the entire pool of decoys, and "single-model" predictors, which evaluate a structure in isolation. This hybrid approach allows for robust ranking when a large pool is available and absolute scoring when it is not. The system utilizes a two-stage training strategy where initial predictor outputs are refined by a second deep learning model to produce the final quality score.
Key Insights: The Power of Distance Features
A significant finding of this work is that continuous distance predictions provide a much richer signal for deep learning than binary contact maps. By analyzing feature importance through SHAP (SHapley Additive exPlanations), the researchers confirmed that distance-based features are among the most influential inputs for the models.
The use of image-processing metrics like SSIM proved particularly effective. These metrics are designed to detect distortions in spatial geometry, making them ideal for identifying "blurry" or physically unrealistic protein models. The "wisdom of the crowd" remains a powerful signal, as evidenced by the high importance of the pairwise consensus score, but distance features provide the necessary granularity to refine these rankings.
Performance in CASP14
The MULTICOM suite demonstrated exceptional performance during the blind CASP14 competition:
- Top Rankings: The three multi-model predictors swept the top three spots for GDT-TS loss out of 68 entries.
- Quantitative Success: MULTICOM-CONSTRUCT achieved an average GDT-TS loss of 0.073, with a high Pearson correlation (~0.88) between predicted and actual quality scores.
- Single-Model Efficacy: MULTICOM-DEEP ranked within the top 10 of single-model methods, proving that distance features enhance accuracy even without consensus data.
Conclusion and Future Directions
While CASP14 marked a significant success for MULTICOM, the emergence of highly accurate folding methods like AlphaFold2 presents new challenges. Future EMA models will need larger training datasets that include high-quality decoys from modern distance-based folding methods to better reflect the current state of the art. Additionally, moving toward end-to-end deep learning—where models learn features directly from 3D coordinates rather than relying on expert-curated metrics—represents the next frontier in protein quality assessment.
Summary updated