TL;DR
DisProtBench is a comprehensive, disorder-aware benchmark designed to evaluate protein structure prediction models (PSPMs) within biologically realistic contexts. By stratifying model evaluation by intrinsic disorder and testing downstream performance on protein–protein interactions (PPI) and drug-binding affinity, DisProtBench demonstrates that disordered regions critically impair extended interfaces while leaving localized binding pockets relatively unscathed.
What Problem Does This Paper Address?
Deep learning models like AlphaFold2, AlphaFold3, and ESMFold have largely solved the static folding problem for well-ordered protein domains. However, real biology relies heavily on intrinsically disordered regions (IDRs)—flexible segments lacking a fixed 3D conformation that mediate signal transduction, transcriptional regulation, and transient interactions.
Current community benchmarks either evaluate static single-chain folding accuracy (e.g., CASP) or treat disorder as a 1D binary sequence classification problem (e.g., CAID). Neither framework evaluates whether 3D structure predictors produce biologically reliable representations when disorder is present at functional interfaces. As a result, researchers cannot easily determine how structural uncertainty impacts downstream biomedical tasks like drug discovery or disease variant analysis.
Methods
DisProtBench introduces a three-tiered evaluation paradigm:
- Data Level: Curates standardized 3D graph representations across disordered disease proteins, GPCR–ligand pairs, and disorder-mediated multimeric complexes.
- Task Level: Evaluates 11 prominent PSPMs (including AlphaFold2, AlphaFold3, Boltz-1, Chai-1, ESMFold, OpenFold, Proteinx, and UniFold) across downstream applications:
- PPI Prediction: Protein complex structures are mapped into volumetric/distance tensors and classified using CNNs (ResNet/DenseNet) to evaluate interface recovery.
- Drug Discovery: Protein–ligand pairs are processed via an Evoformer-based architecture (LISA-CPI) to predict compound binding affinity.
- Disorder Stratification via pLDDT: The authors demonstrate that residue-level confidence (pLDDT) exhibits a sharp bimodal distribution separating disordered residues (peaking around 30–40) from ordered residues (peaking above 85). They operationalize this by benchmarking models under three regimes: full sequence, $\text{pLDDT} \ge 30$, and $\text{pLDDT} \ge 50$.
- User Level (Interactive Portal): A web-based visual analysis suite providing precomputed 3D structures, residue-level B-factor overlays, and RMSD alignment heatmaps for interactive error diagnosis.
Datasets
The benchmark unifies three curated datasets:
- Disorder in Human Disease Dataset: UniProt proteins with long disordered segments ($\ge 20$ residues), paired with high-confidence interaction partners from HINT and clinical missense variants from ClinVar ($\ge 2$ pathogenic mutations per gene).
- Individual Protein (GPCR & Ligand) Dataset: 71,757 GPCR–ligand pairs and 33,212 pain-related pairs from ChEMBL and BindingDB ($pK_i \in [6, 9]$), annotated with agonist/antagonist pharmacology, FDA-approved drugs, and gut metabolites.
- Disorder-Mediated PPI Dataset: Multimeric complexes generated from AlphaFold-Multimer, filtered for quality ($\text{median pLDDT} \ge 70$, interface compactness) and IDR-mediated contact interfaces.
Key Insights and Ideas
- Disorder selectively degrades interface tasks: Removing disordered regions ($\text{pLDDT} \ge 50$) significantly improves PPI interface recall and classification F1 scores, proving that conformational ambiguity in IDRs directly misleads downstream interaction modeling.
- Pocket localization buffers small-molecule binding: In contrast to PPIs, ligand-binding affinity predictions in drug discovery tasks are remarkably stable across pLDDT thresholds. Localized binding pockets tend to remain structured even when flanking protein segments are disordered.
- pLDDT is a robust standalone disorder proxy: Evaluating composite metrics that integrate PAE (predicted aligned error) or actifpTM alongside pLDDT revealed no notable performance gain over simple pLDDT filtering, verifying pLDDT's utility as a scalable surrogate for structural disorder.
- Global metrics mask critical regional vulnerabilities: High global fold scores do not guarantee reliable interface geometry, demonstrating the necessity of function-oriented, disorder-aware benchmarks.
Key Results
- PPI Prediction:
- AlphaFold3 achieved the highest overall PPI classification performance ($\text{F1} = 0.901$ on full sequence, rising to $0.914$ at $\text{pLDDT} \ge 50$).
- Boltz-1 and Chai-1 closely trailed AF3 ($\text{F1} \approx 0.869\text{--}0.871$ at $\text{pLDDT} \ge 50$), demonstrating high robustness to structural ambiguity.
- Open-source Evoformer implementations struggled on flexible interfaces; OpenFold and UniFold achieved F1 scores of only $0.653$ and $0.526$, respectively, even in high-confidence regions ($\text{pLDDT} \ge 50$).
- Drug Discovery:
- AlphaFold3 led prediction accuracy ($\text{MAE} = 0.048$, Pearson $R \approx 0.999$, MSE = $0.0040$), with Boltz, Chai-1, and ESMFold showing competitive accuracy across confidence cutoffs.
Limitations and Future Work
- Static Proxies vs. Ensembles: The benchmark relies primarily on single static predictions and confidence metrics (pLDDT) as proxies for disorder, rather than evaluating conformational ensembles generated via molecular dynamics or generative sampling.
- Future Directions: The authors plan to extend DisProtBench to support coarse-grained and ensemble predictions, integrate additional disorder repositories (e.g., MobiDB), provide real-time interpretability tools, and expand tasks to complex disorder-driven phenomena like liquid–liquid phase separation.
Summary updated