Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Julia Koehler Leman, Sergey Lyskov, Steven M. Lewis, Jared Adolf-Bryfogle, Rebecca F. Alford, Kyle Barlow, Ziv Ben-Aharon, Daniel Farrell, Jason Fell, William A. Hansen, Ameya Harmalkar, Jeliazko Jeliazkov, Georg Kuenze, Justyna D. Krys, Ajasja Ljubetič, Amanda L. Loshbaugh, Jack Maguire, Rocco Moretti, Vikram Khipple Mulligan, Morgan L. NancePhuong T. Nguyen, Shane Ó Conchúir, Shourya S. Roy Burman, Rituparna Samanta, Shannon T. Smith, Frank Teets, Johanna K.S. Tiemann, Andrew Watkins, Hope Woods, Brahm J. Yachnin, Christopher D. Bahl, Chris Bailey-Kellogg, David Baker, Rhiju Das, Frank DiMaio, Sagar D. Khare, Tanja Kortemme, Jason W. Labonte, Kresten Lindorff-Larsen, Jens Meiler, William Schief, Ora Schueler-Furman, Justin B. Siegel, Amelie Stein, Vladimir Yarov-Yarovoy, Brian Kuhlman, Andrew Leaver-Fay, Dominik Gront, Jeffrey J. Gray, Richard Bonneau

Research output: Contribution to journalArticlepeer-review

11 Scopus citations


Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

Original languageEnglish (US)
Article number6947
JournalNature communications
Issue number1
StatePublished - Dec 2021

All Science Journal Classification (ASJC) codes

  • General Chemistry
  • General Biochemistry, Genetics and Molecular Biology
  • General Physics and Astronomy


Dive into the research topics of 'Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks'. Together they form a unique fingerprint.

Cite this