Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Julia Koehler Leman, Sergey Lyskov, Steven M. Lewis, Jared Adolf‐Bryfogle, Rebecca F. Alford, Kyle A. Barlow, Ziv Ben-Aharon, Daniel P. Farrell, Jason S. Fell, William A. Hansen, Ameya Harmalkar, Jeliazko R. Jeliazkov, Georg Kuenze, Justyna Krys, Ajasja Ljubetič, Amanda L. Loshbaugh, Jack B. Maguire, Rocco Moretti, Vikram Khipple Mulligan, Phuong T. Nguyen, Shane Ó Conchúir, Shourya S. Roy Burman, Shannon T. Smith, Frank D. Teets, Johanna K. S. Tiemann, Andrew M. Watkins, Hope Woods, Brahm J. Yachnin, Christopher D. Bahl, Chris Bailey‐Kellogg, David Baker, Rhiju Das, Frank DiMaio, Sagar D. Khare, Tanja Kortemme, Jason W. Labonte, Kresten Lindorff‐Larsen, Jens Meiler, William R. Schief, Ora Schueler‐Furman, Justin B. Siegel, Amelie Stein, Vladimir Yarov‐Yarovoy, Brian Kuhlman, Andrew Leaver‐Fay, Dominik Gront, Jeffrey J. Gray, Richard Bonneau

Abstract

Abstract Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.
BibTeX
@inproceedings{leman2021ensuring,
  author = {Julia Koehler Leman and Sergey Lyskov and Steven M. Lewis and Jared Adolf‐Bryfogle and Rebecca F. Alford and Kyle A. Barlow and Ziv Ben-Aharon and Daniel P. Farrell and Jason S. Fell and William A. Hansen and Ameya Harmalkar and Jeliazko R. Jeliazkov and Georg Kuenze and Justyna Krys and Ajasja Ljubetič and Amanda L. Loshbaugh and Jack B. Maguire and Rocco Moretti and Vikram Khipple Mulligan and Phuong T. Nguyen and Shane Ó Conchúir and Shourya S. Roy Burman and Shannon T. Smith and Frank D. Teets and Johanna K. S. Tiemann and Andrew M. Watkins and Hope Woods and Brahm J. Yachnin and Christopher D. Bahl and Chris Bailey‐Kellogg and David Baker and Rhiju Das and Frank DiMaio and Sagar D. Khare and Tanja Kortemme and Jason W. Labonte and Kresten Lindorff‐Larsen and Jens Meiler and William R. Schief and Ora Schueler‐Furman and Justin B. Siegel and Amelie Stein and Vladimir Yarov‐Yarovoy and Brian Kuhlman and Andrew Leaver‐Fay and Dominik Gront and Jeffrey J. Gray and Richard Bonneau},
  title = {{Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks}},
  booktitle = {Nature Communications},
  year = {2021},
  doi = {10.1038/s41467-021-27222-7},
}

Loading related content...

Figure