Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Charles R. Ebersole, Maya B. Mathur, Erica Baranski, Diane-Jo Bart-Plange, Nicholas R. Buttrick, Christopher R. Chartier, Katherine S. Corker, Martin Corley, Joshua K. Hartshorne, Hans IJzerman, Ljiljana B. Lazarević, Hugh Rabagliati, Ivan Ropovik, Balázs Aczél, Lena Fanya Aeschbach, Luca Andrighetto, Jack Arnal, Holly Arrow, Peter Babinčák, Bence E. Bakos, Gabriel Baník, Ernest Baskin, Radomir Belopavlović, Michael H. Bernstein, Michał Białek, Nicholas Bloxsom, Bojana Bodroža, Diane B. V. Bonfiglio, Leanne Boucher, Florian Brühlmann, Claudia Chloe Brumbaugh, Erica Casini, Yiling Chen, Carlo Chiorri, William J. Chopik, Oliver Christ, Antonia M. Ciunci, Heather M. Claypool, Sean P. Coary, Marija V. Čolić, W. Matthew Collins, Paul Curran, Chris Day, Benjamin Dering, Anna Dreber, John E. Edlund, Filipe Falcão, Anna Fedor, Lily Feinberg, Ian Ferguson, Máire B. Ford, Michael C. Frank, Emily Fryberger, Alexander Garinther, Katarzyna Gawryluk, Kayla Ashbaugh, Mauro Giacomantonio, Steffen R. Giessner, Jon Grahe, Rosanna E. Guadagno, Ewa Hałasa, Peter Hancock, Rias A. Hilliard, Joachim Hüffmeier, Sean Hughes, Katarzyna Idzikowska, Michael Inzlicht, Alan Jern, William Jiménez‐Leal, Magnus Johannesson, Jennifer A. Joy-Gaba, Mathias Kauff, Danielle Kellier, Grecia Kessinger, Mallory C. Kidwell, Amanda M. Kimbrough, Josiah King, Vanessa S. Kolb, Sabina Kołodziej, Márton Kovács, Karolina Krasuska, Sue Kraus, Lacy E. Krueger, Katarzyna Kuchno, Caio Ambrosio Lage, Eleanor V. Langford, Carmel Levitan, Tiago Jessé Souza de Lima, Hause Lin, Samuel Lins, Jia E. Loy, Dylan Manfredi, Łukasz Markiewicz, Madhavi Menon, Brett Mercier, Mitchell M. Metzger, Venus Meyet, Ailsa E. Millen, Jeremy K. Miller, Andres MontealegreAdvances 2020 · DOI

Abstract

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).
BibTeX
@inproceedings{ebersole2020many,
  author = {Charles R. Ebersole and Maya B. Mathur and Erica Baranski and Diane-Jo Bart-Plange and Nicholas R. Buttrick and Christopher R. Chartier and Katherine S. Corker and Martin Corley and Joshua K. Hartshorne and Hans IJzerman and Ljiljana B. Lazarević and Hugh Rabagliati and Ivan Ropovik and Balázs Aczél and Lena Fanya Aeschbach and Luca Andrighetto and Jack Arnal and Holly Arrow and Peter Babinčák and Bence E. Bakos and Gabriel Baník and Ernest Baskin and Radomir Belopavlović and Michael H. Bernstein and Michał Białek and Nicholas Bloxsom and Bojana Bodroža and Diane B. V. Bonfiglio and Leanne Boucher and Florian Brühlmann and Claudia Chloe Brumbaugh and Erica Casini and Yiling Chen and Carlo Chiorri and William J. Chopik and Oliver Christ and Antonia M. Ciunci and Heather M. Claypool and Sean P. Coary and Marija V. Čolić and W. Matthew Collins and Paul Curran and Chris Day and Benjamin Dering and Anna Dreber and John E. Edlund and Filipe Falcão and Anna Fedor and Lily Feinberg and Ian Ferguson and Máire B. Ford and Michael C. Frank and Emily Fryberger and Alexander Garinther and Katarzyna Gawryluk and Kayla Ashbaugh and Mauro Giacomantonio and Steffen R. Giessner and Jon Grahe and Rosanna E. Guadagno and Ewa Hałasa and Peter Hancock and Rias A. Hilliard and Joachim Hüffmeier and Sean Hughes and Katarzyna Idzikowska and Michael Inzlicht and Alan Jern and William Jiménez‐Leal and Magnus Johannesson and Jennifer A. Joy-Gaba and Mathias Kauff and Danielle Kellier and Grecia Kessinger and Mallory C. Kidwell and Amanda M. Kimbrough and Josiah King and Vanessa S. Kolb and Sabina Kołodziej and Márton Kovács and Karolina Krasuska and Sue Kraus and Lacy E. Krueger and Katarzyna Kuchno and Caio Ambrosio Lage and Eleanor V. Langford and Carmel Levitan and Tiago Jessé Souza de Lima and Hause Lin and Samuel Lins and Jia E. Loy and Dylan Manfredi and Łukasz Markiewicz and Madhavi Menon and Brett Mercier and Mitchell M. Metzger and Venus Meyet and Ailsa E. Millen and Jeremy K. Miller and Andres Montealegre},
  title = {{Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability}},
  booktitle = {Advances},
  year = {2020},
  doi = {10.1177/2515245920958687},
}

Loading related content...

Figure