Abstract
Intrinsically disordered regions (IDRs) play central roles in cellular function, yet remain poorly evaluated by existing protein structure prediction benchmarks. Current evaluations largely focus on well-folded domains, overlooking three fundamental challenges in realistic biological settings: the structural complexity of proteins, the resulting low availability of reliable ground truth, and prediction uncertainty that can propagate into high-risk downstream failures. We present DisProtBench, an IDR-centric benchmark that explicitly incorporates prediction uncertainty into the evaluation of protein structure prediction models (PSPMs). To address structural complexity and ground-truth scarcity, we curate and unify a large-scale, multi-modal dataset spanning disease-relevant IDRs, GPCR-ligand interactions, and multimeric protein complexes. To assess predictive uncertainty, we introduce Functional Uncertainty Sensitivity (FUS), a novel prediction uncertainty-stratified metric that quantifies downstream task performance under prediction uncertainty. Using this benchmark, we conduct a systematic evaluation of state-of-the-art PSPMs and reveal clear, task-dependent failure modes. Protein-protein interaction prediction degrades sharply in IDRs, while structure-based drug discovery remains comparatively robust. These effects are largely invisible to standard global accuracy metrics, which overestimate functional reliability under prediction uncertainty. We have open-sourced our benchmark and the codebase at https://github.com/Susan571/DisProtBench.
Introduction: The Challenge of Intrinsically Disordered Regions
Deep learning models for Protein Structure Prediction (PSPMs) have revolutionized biology, achieving near-atomic accuracy on well-folded proteins. However, existing evaluation frameworks, such as CASP, typically assume proteins possess a single, static 3D structure. In reality, many crucial proteins contain Intrinsically Disordered Regions (IDRs)—flexible segments that lack a fixed shape and play vital roles in cellular signaling and regulation.
Evaluating IDRs presents three core challenges: structural complexity across multiple scales, low availability of reliable ground truth data, and the propagation of prediction uncertainty into high-stakes downstream applications like drug discovery. Standard model confidence scores, such as AlphaFold's pLDDT, often fail to indicate whether a "floppy" prediction will lead to catastrophic failure in a specific functional context.
Methodology: Evaluating Functional Reliability
DisProtBench shifts the focus from raw structural accuracy to functional reliability. Instead of measuring how well a predicted structure matches an often non-existent ground truth, the benchmark evaluates how predicted structures perform in downstream biological tasks.
Functional Uncertainty Sensitivity (FUS)
A key contribution of this work is the Functional Uncertainty Sensitivity (FUS) metric. FUS measures how downstream task performance changes as structurally uncertain regions are progressively filtered out. This diagnostic tool identifies whether uncertainty is functionally harmful: if removing high-uncertainty regions significantly improves performance, that uncertainty was a limiting factor for the task.
Multi-modal Datasets
To capture IDR complexity, we curated three large-scale datasets:
- DisProt-Based Dataset: ~1,000 human proteins with long IDRs and disease-associated variants, linked to ~10,000 protein-protein interactions (PPIs).
- Individual Protein Dataset: ~100,000 GPCR-ligand interactions for structure-based drug discovery.
- Protein Interaction Dataset: ~1,000 multimeric complexes with flexible, IDR-mediated interfaces.
Results and Discussion
The evaluation of 11 state-of-the-art models (including AlphaFold 2/3, ESMFold, and Boltz) revealed that structural uncertainty's impact is highly task-dependent.
Task-Dependent Robustness
Protein-Protein Interaction (PPI) prediction relies heavily on flexible interfaces; consequently, IDR uncertainty significantly degrades performance. For example, filtering out uncertain regions (FUS threshold $\tau=50$) improved AlphaFold2's F1 score from 0.791 to 0.817. Conversely, drug discovery tasks, which depend on compact, well-structured binding pockets, proved remarkably robust to IDR noise elsewhere in the protein. AlphaFold2's Mean Absolute Error (MAE) for drug binding shifted negligibly from 0.173 to 0.164 under filtering.
Limitations of Standard Metrics
Our results demonstrate that standard global accuracy metrics are often blind to functional risk. Two proteins may share identical average confidence scores but exhibit vastly different downstream success rates depending on the spatial distribution of uncertainty. Models that appear competitive under aggressive confidence filtering may still fail in the uncertainty-rich regimes typical of real-world biological interactions.
Conclusion
DisProtBench provides a necessary framework for evaluating PSPMs in the context of protein flexibility. While newer models like Boltz and Chai show improved robustness to interface geometry uncertainty, the field must move toward dynamic or ensemble structural representations. Future work will explore alternative uncertainty estimates and additional downstream tasks, such as phase separation, to further uncover the functional limits of protein structure prediction.
Summary updated