TL;DR
The CASP16 challenge assessed computational methods for predicting how drug-like molecules bind to proteins (pose) and how strongly (affinity). While template-based methods performed well for pose prediction, AlphaFold 3 surprisingly outperformed all human participants. Affinity prediction accuracy remains a significant challenge, limited by current scoring functions.
What Problem Does This Paper Address?
Predicting how small molecules (ligands) bind to proteins is a cornerstone of computer-aided drug design (CADD). Accurately determining both the 3D orientation of the ligand within the protein's binding site (pose) and the strength of that interaction (affinity) is crucial for designing effective drugs. However, achieving high accuracy in these predictions, especially for novel drug-like compounds, remains a significant hurdle. The CASP (Critical Assessment of Structure Prediction) initiative, known for evaluating protein structure prediction, expanded its scope to include protein-ligand interactions in CASP15 and further refined this in CASP16 by focusing specifically on drug-like molecules from pharmaceutical projects. This paper presents the results of the CASP16 protein-ligand pose and affinity prediction challenge, highlighting the state-of-the-art and persistent challenges in this field.
Methods
The CASP16 challenge involved participants submitting predictions for protein-ligand complexes. For pose prediction, the primary metric was the LDDT-PLI score, which measures the accuracy of ligand atom positions relative to the crystal structure. For affinity prediction, Kendall's τ statistic was used to assess the ranking power of predicted affinities against experimental data. The study also included baseline predictors, such as AlphaFold 3, Boltz-1, RosettaFold All-Atom (deep learning methods), and AutoDock Vina (a docking method), to provide a benchmark for participant performance. Notably, the challenge included a "Stage 2" where participants were given experimental structures to improve their affinity predictions, allowing assessment of whether structural information alone could boost accuracy.
Datasets
The CASP16 challenge utilized datasets derived from pharmaceutical drug discovery projects, focusing on five protein systems: Chymase, Cathepsin G, Autotaxin, SARS-CoV-2 main protease (Mpro), and WDR55. These datasets included 229 protein-ligand pose targets and 140 affinity targets. The ligands were specifically chosen to be "drug-like" compounds. This differs from previous CASP challenges that often used naturally occurring molecules or less drug-focused compounds.
Key Insights and Ideas
- Template-based methods still shine for pose prediction: Despite the rise of deep learning, methods that leverage existing similar protein-ligand structures in the PDB performed very well, indicating the continued importance of structural homology.
- AlphaFold 3 is a powerhouse: The automated AlphaFold 3 system significantly outperformed all human participants in pose prediction, suggesting that advanced deep learning models can surpass human-guided approaches when trained on vast datasets.
- Affinity prediction remains a bottleneck: The correlation between predicted and experimental affinities was modest, and providing experimental structures in Stage 2 did not significantly improve predictions. This points to limitations in current scoring functions as the primary bottleneck.
- Ligand complexity isn't a simple predictor of accuracy: Surprisingly, simple metrics like ligand size or flexibility showed minimal correlation with pose prediction accuracy.
- Reliability assessment is challenging: Participants struggled to accurately estimate the reliability of their pose predictions, with low correlations between their self-assessed scores (LScore) and actual accuracy.
Key Results
- Pose Prediction: The best CASP16 participants achieved a mean LDDT-PLI of 0.69. AlphaFold 3 achieved a mean LDDT-PLI of 0.8, outperforming the best human participants.
- Affinity Prediction: The maximum Kendall's τ achieved by participants was 0.42, which is significantly below the theoretical maximum (~0.73) and similar to results from previous challenges like D3R.
- Stage 2 Impact: Providing experimental structures in Stage 2 did not improve affinity prediction accuracy, reinforcing the idea that scoring functions are the limiting factor.
- Baseline Performance: AlphaFold 3 was the top performer among baseline methods for pose prediction. For affinity prediction, simple descriptors like molecular weight showed some correlation, but more sophisticated methods like Gaussian Process Regression and docking-based approaches had mixed results.
Limitations and Future Work
The authors acknowledge that the performance of methods can vary depending on the specific targets, and a different set of targets might lead to a different ranking of participants. They also note that the challenge primarily focused on well-characterized protein targets, and future challenges should include less well-characterized systems to better assess method generalizability.
Future work should focus on:
- Developing more accurate scoring functions for affinity prediction.
- Exploring simulation-based free energy methods, which are computationally intensive but can be highly accurate.
- Investigating methods for better assessing the reliability of pose predictions.
- Including a wider range of computational approaches, particularly those used in industrial drug discovery.
- Incorporating less well-characterized target systems to test method generalizability.
Summary updated