An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12
Chen Keasar, Liam J. McGuffin, Björn Wallner, Gaurav Chopra, Badri Adhikari, Debswapna Bhattacharya, Lauren A. Blake, Leandro Oliveira Bortot, Renzhi Cao, B. K. Dhanasekaran, Itzhel Dimas, Rodrigo Antonio Faccioli, Eshel Faraggi, Robert Ganzynkowicz, Sambit Ghosh, Soma Ghosh, Artur Giełdoń, Łukasz Golon, Yi He, Lim Heo, Jie Hou, Main Khan, Firas Khatib, George Khoury, Chris A. Kieslich, David E. Kim, Paweł Krupa, Gyu Rie Lee, Hongbo Li, Jilong Li, Agnieszka G. Lipska, Adam Liwo, Ali H. A. Maghrabi, Milot Mirdita, Shokoufeh Mirzaei, Magdalena A. Mozolewska, Melis Onel, Sergey Ovchinnikov, Anand S. Shah, Utkarsh Shah, Tomer Sidi, Adam K. Sieradzan, Magdalena J. Ślusarz, Rafał Ślusarz, James B. Smadbeck, Phanourios Tamamis, Nicholas Trieber, Tomasz Wirecki, Yanping Yin, Yang Zhang, Jaume Bacardit, Maciej Baranowski, Nicholas Chapman, Seth Cooper, Alexandre Defelicibus, Jeff Flatten, Brian Koepnick, Zoran Popović, Bartłomiej Zaborowski, David Baker, Jianlin Cheng, Cezary Czaplewski, Alexandre C. B. Delbem, Christodoulos A. Floudas, Andrzej Kloczkowski, Stanisław Ołdziej, Michael Levitt, Harold A. Scheraga, Chaok Seok, Johannes Söding, Saraswathi Vishveshwara, Dong Xu, Ahmet Çağlar, Alan Coral, Alison MacMillan, Allen Lubow, Barbara Failer, Bruno Kestemont, Catherine R. Landers, Charles Robert Painter, Christophe Garnier, Claudine Sellin, Dietmar Janz, Douglas Craig Wheeler, Vera Simon, Dylan M. Flear, Emmanuel Croze, George Victor McIlvaine, Gil Beecher, Gordon Lawrie, Guy Ykman, Harald Feldmann, Heidemarie K. Fuentes, Hosokawa Terumasa, Istvan Kovanecz, James Absalom Longino, Jan Hendrik Nijland, Jasper A. Diderich, Jeffrey M. Canfield, Jesper ErikssonScientific2018 · DOI
The WeFold collaborative successfully leveraged a "coopetition" model to build diverse protein structure prediction pipelines for CASP11 and CASP12, demonstrating improved performance and creating a valuable public resource. This approach highlights the power of community-driven efforts in tackling complex scientific challenges.
What Problem Does This Paper Address?
Predicting the 3D structure of a protein from its amino acid sequence is a fundamental challenge in biology with vast implications for understanding disease and designing drugs. While the Critical Assessment of Protein Structure Prediction (CASP) experiments have driven progress, the field faces hurdles like the multi-step nature of prediction pipelines and a high barrier to entry for new ideas. Individual labs struggle to develop comprehensive pipelines, and integrating diverse methods is difficult due to non-standard interfaces. WeFold was created to address this by fostering collaboration and allowing researchers to share and combine their specialized tools into hybrid pipelines.
Methods
WeFold operates as a "coopetition" – a blend of cooperation and competition. Participating groups contribute individual components (e.g., decoy generation, contact prediction, quality assessment, refinement) to build hybrid pipelines. These pipelines then compete in CASP, allowing for objective evaluation of their integrated performance. The paper analyzes data from CASP11 (WeFold2) and CASP12 (WeFold3), comparing the performance of these hybrid pipelines against each other and against established CASP groups. The analysis focuses on metrics like GDT_TS (Global Distance Test Total Score), a standard measure of structural similarity.
Datasets
The primary datasets used are the protein targets provided by the CASP experiments for CASP11 and CASP12. The WeFold pipelines generate and submit predicted protein models (decoys) for these targets. The paper also mentions the availability of all contributed and generated models on the WeFold website, creating a valuable resource for future research.
Key Insights and Ideas
The Power of "Coopetition": WeFold demonstrates that combining individual expertise into collaborative pipelines can achieve results beyond what any single lab could accomplish, pushing the boundaries of what's possible in protein structure prediction.
Bridging the Gap: By lowering the entry barrier and encouraging the integration of diverse methods, WeFold successfully attracted researchers from various fields, fostering innovation.
WeFold3 Improvement: The study shows a clear improvement in WeFold3's performance compared to WeFold2, indicating that lessons learned from the earlier experiment were effectively implemented.
Bottlenecks in Prediction: The analysis highlights that while individual components are improving, bottlenecks remain in areas like clustering, filtering, and quality assessment, especially when dealing with a large and diverse set of models.
Data-Driven Science: The vast amount of data generated by WeFold is recognized as a significant resource for developing new methods and advancing the field, moving towards a more data-science-oriented approach.
Key Results
Improved Performance: WeFold3 pipelines showed a larger proportion of "best or very-close-to-best" decoys compared to WeFold2, with twice as many WeFold3 models matching the best CASP decoy (Δ=0) compared to WeFold2.
CASP Ranking: While no WeFold pipelines ranked highly in CASP11, eight WeFold3 pipelines achieved top rankings in CASP12.
Specific Pipeline Successes: Several WeFold3 pipelines, particularly those based on MESHI selection (wfMESHI-TIGRESS, wfMESHI-Seok) and some Rosetta-based pipelines (wfRosetta-ProQ-ModF6, wfAll-Cheng), showed competitive performance, with some even outperforming established groups in specific categories.
Refinement Gains: In refinement tasks, Foldit players consistently improved starting models, with some targets showing significant GDT_HA improvements.
Limitations and Future Work
Pipeline Assembly Timing: A key limitation is that full pipelines are assembled on the first day of CASP without prior benchmarking, potentially leading to suboptimal configurations compared to heavily benchmarked individual group methods.
QA Method Generalization: Many Quality Assessment (QA) methods are trained on specific types of models (e.g., server models) and struggle to generalize to the broader range of models generated by a single group or pipeline.
Domain Splitting Challenges: For Rosetta-based pipelines, domain assembly and splitting proved to be an obstacle, impacting performance, especially in Free Modeling categories.
Future Directions: The authors suggest efforts to improve sampling, domain splitting, decoy set reduction, and QA/selection algorithms. They also highlight the potential of leveraging the "big data" generated by WeFold for future advancements and plan to make pipeline codes publicly available.
Summary updated
Abstract
Every two years groups worldwide participate in the Critical Assessment of Protein Structure Prediction (CASP) experiment to blindly test the strengths and weaknesses of their computational methods. CASP has significantly advanced the field but many hurdles still remain, which may require new ideas and collaborations. In 2012 a web-based effort called WeFold, was initiated to promote collaboration within the CASP community and attract researchers from other fields to contribute new ideas to CASP. Members of the WeFold coopetition (cooperation and competition) participated in CASP as individual teams, but also shared components of their methods to create hybrid pipelines and actively contributed to this effort. We assert that the scale and diversity of integrative prediction pipelines could not have been achieved by any individual lab or even by any collaboration among a few partners. The models contributed by the participating groups and generated by the pipelines are publicly available at the WeFold website providing a wealth of data that remains to be tapped. Here, we analyze the results of the 2014 and 2016 pipelines showing improvements according to the CASP assessment as well as areas that require further adjustments and research.
BibTeX
@inproceedings{keasar2018analysis,
author = {Chen Keasar and Liam J. McGuffin and Björn Wallner and Gaurav Chopra and Badri Adhikari and Debswapna Bhattacharya and Lauren A. Blake and Leandro Oliveira Bortot and Renzhi Cao and B. K. Dhanasekaran and Itzhel Dimas and Rodrigo Antonio Faccioli and Eshel Faraggi and Robert Ganzynkowicz and Sambit Ghosh and Soma Ghosh and Artur Giełdoń and Łukasz Golon and Yi He and Lim Heo and Jie Hou and Main Khan and Firas Khatib and George Khoury and Chris A. Kieslich and David E. Kim and Paweł Krupa and Gyu Rie Lee and Hongbo Li and Jilong Li and Agnieszka G. Lipska and Adam Liwo and Ali H. A. Maghrabi and Milot Mirdita and Shokoufeh Mirzaei and Magdalena A. Mozolewska and Melis Onel and Sergey Ovchinnikov and Anand S. Shah and Utkarsh Shah and Tomer Sidi and Adam K. Sieradzan and Magdalena J. Ślusarz and Rafał Ślusarz and James B. Smadbeck and Phanourios Tamamis and Nicholas Trieber and Tomasz Wirecki and Yanping Yin and Yang Zhang and Jaume Bacardit and Maciej Baranowski and Nicholas Chapman and Seth Cooper and Alexandre Defelicibus and Jeff Flatten and Brian Koepnick and Zoran Popović and Bartłomiej Zaborowski and David Baker and Jianlin Cheng and Cezary Czaplewski and Alexandre C. B. Delbem and Christodoulos A. Floudas and Andrzej Kloczkowski and Stanisław Ołdziej and Michael Levitt and Harold A. Scheraga and Chaok Seok and Johannes Söding and Saraswathi Vishveshwara and Dong Xu and Ahmet Çağlar and Alan Coral and Alison MacMillan and Allen Lubow and Barbara Failer and Bruno Kestemont and Catherine R. Landers and Charles Robert Painter and Christophe Garnier and Claudine Sellin and Dietmar Janz and Douglas Craig Wheeler and Vera Simon and Dylan M. Flear and Emmanuel Croze and George Victor McIlvaine and Gil Beecher and Gordon Lawrie and Guy Ykman and Harald Feldmann and Heidemarie K. Fuentes and Hosokawa Terumasa and Istvan Kovanecz and James Absalom Longino and Jan Hendrik Nijland and Jasper A. Diderich and Jeffrey M. Canfield and Jesper Eriksson},
title = {{An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12}},
booktitle = {Scientific},
year = {2018},
doi = {10.1038/s41598-018-26812-8},
}