DISPROTBENCH: A Disorder-Aware, Task-Rich Benchmark for Evaluating Protein Structure Prediction in Realistic Biological Contexts

Xinyue Zeng, Tuo Wang, Adithya Kulkarni, Alexander Lu, Alexandra Ni, Phoebe Xing, Junhan Zhao, Siwei Chen, Dawei Zhou

Introduction

Structural biology has traditionally focused on "folded" proteins—stable, well-defined 3D architectures. AI breakthroughs like AlphaFold and ESMFold have achieved remarkable success in predicting these static shapes. However, a significant portion of the human proteome consists of Intrinsically Disordered Regions (IDRs). Unlike folded domains, IDRs lack a single stable conformation; they exist as dynamic ensembles that adapt to their biological environment.

Conceptual representation of protein states: disordered, flexible ensembles, and ordered structures.

Current evaluation benchmarks, such as CASP, often treat proteins as static objects. When AI models predict structures for IDRs, they frequently report high uncertainty. Standard metrics often overlook this uncertainty or fail to quantify its impact on downstream functional applications like drug discovery or protein-protein interaction (PPI) modeling. This creates a significant gap in our understanding of the "dark matter" of the proteome.

The DisProtBench Framework

To address these gaps, the authors present DisProtBench, a comprehensive framework designed to evaluate protein structure prediction models through the lens of functional reliability rather than mere geometric accuracy. DisProtBench focuses on three core challenges: structural complexity, low availability of reliable ground truth, and prediction uncertainty.

Overview of DisProtBench: Core challenges in IDR evaluation and the benchmark's multi-faceted approach to addressing them.

The framework introduces several key components:

  1. Unified Multi-Modal Data Curation: Integration of datasets covering disease-relevant IDRs, GPCR-ligand interactions, and multimeric complexes.
  2. Functional Uncertainty Sensitivity (FUS): A novel metric that measures how a model's performance on specific biological tasks fluctuates when residues are filtered based on their predicted uncertainty.
  3. Visual Analytics Portal: An interactive interface for side-by-side comparison of models (e.g., AF2, AF3, Boltz) to identify where structural uncertainty translates into functional failure.

Datasets and Methodology

DisProtBench utilizes three specialized datasets to probe different failure modes:

  • DisProt-Based Dataset: Contains ~1,000 proteins and ~10,000 interactions, focusing on long IDRs and disease-associated variants.
  • Individual Protein Dataset: Comprises ~100,000 GPCR-ligand interactions to evaluate structure-based drug discovery.
  • Protein Interaction Dataset: Includes ~1,000 complexes mediated by flexible IDR interfaces.

Key Insights: Task Sensitivity and FUS

A primary finding of the study is that structural uncertainty impacts biological tasks differently. Protein-Protein Interaction (PPI) prediction is highly sensitive; even minor uncertainty in an IDR can lead to complete failure in binding site prediction. In contrast, Drug Discovery (ligand binding) appears more robust. This is likely because drug targets often involve stable, well-defined pockets, whereas PPIs rely on large, flexible surface areas where conformational "wobble" is critical.

The study also demonstrates that the standard confidence metric, pLDDT, is often insufficient for disordered regions. The proposed FUS metric aligns much more closely with ground-truth biological data ($\rho = 0.74$ for FUS vs. $\rho = -0.48$ for mean pLDDT), providing a more reliable diagnostic tool for researchers.

Results and Performance

Evaluation across 11 models, including AlphaFold3 (AF3) and Boltz, revealed that filtering by FUS consistently improves task accuracy. For PPI tasks, AF3's F1 score increased from 0.901 to 0.914 when high-uncertainty regions were removed, indicating that model errors are heavily concentrated in disordered zones. While newer models show general improvements, they still exhibit significant, task-dependent failure modes in IDRs that global accuracy metrics fail to capture.

Conclusion and Future Directions

DisProtBench highlights the need for a paradigm shift in protein AI evaluation. Future work will focus on:

  • Ensemble Modeling: Moving beyond single "average" structures to represent the multi-state nature of IDRs.
  • Expanding Task Breadth: Including complex phenomena like phase separation and cellular signaling.
  • Independent Uncertainty Quantification: Developing methods to identify "confidently wrong" predictions that might bypass current benchmarks.

By providing a "stress test" for structural reliability, DisProtBench offers a path toward more robust AI applications in the most dynamic and challenging regions of biology.

Summary updated

Abstract

Recent advances in protein structure prediction have achieved near-atomic accuracy for well-folded proteins. However, current benchmarks inadequately assess model performance in biologically challenging contexts, especially those involving intrinsically disordered regions (IDRs), limiting their utility in applications like drug discovery, disease variant interpretation, and protein interface design. We introduce DisProtBench, a comprehensive benchmark for evaluating protein structure prediction models (PSPMs) under structural disorder and complex biological conditions. DisProtBench spans three key axes: (1) Data complexity—covering disordered regions, G protein-coupled receptors (GPCR)–ligand pairs, and multimeric complexes; (2) Task diversity—benchmarking twelve leading PSPMs across structure-based tasks with unified classification, regression, and interface metrics; and (3) Interpretability—via the DisProtBench Portal, offering precomputed 3D structures and visual error analyses. Our results reveal significant variability in model robustness under disorder, with low-confidence regions linked to functional prediction failures. Notably, global accuracy metrics often fail to predict task performance in disordered settings, emphasizing the need for function-aware evaluation. DisProtBench establishes a reproducible, extensible, and biologically grounded framework for assessing next-generation PSPMs in realistic biomedical scenarios.
BibTeX
@misc{zeng2025disprotbench,
  author = {Xinyue Zeng and Tuo Wang and Adithya Kulkarni and Alexander Lu and Alexandra Ni and Phoebe Xing and Junhan Zhao and Siwei Chen and Dawei Zhou},
  title = {{DISPROTBENCH: A Disorder-Aware, Task-Rich Benchmark for Evaluating Protein Structure Prediction in Realistic Biological Contexts}},
  year = {2025},
  eprint = {2507.02883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.02883},
}
Figure