TL;DR
This paper proposes a framework for making sound replication decisions by considering the interplay of epistemic (truth-seeking) and nonepistemic (practical, ethical) values, filtered through two distinct cognitive attitudes: the "Book of Truths" and the "Book of Conversations." It argues that aligning replication decisions with these values and attitudes is crucial for scientific progress and public trust.
What Problem Does This Paper Address?
The scientific community is grappling with a perceived "replication crisis," where many published findings fail to hold up when re-examined. This has led to calls for reforms in publishing, methodology, and reporting. However, resources (time, money) for conducting replications are limited, making it essential for researchers to make judicious decisions about whether and how to replicate. This paper addresses the challenge of how scientists can navigate these decisions effectively, moving beyond simply acknowledging the problem to providing a structured approach for making sound choices.
Methods
The authors propose a conceptual framework that integrates scientific values and cognitive attitudes to guide replication decisions. They distinguish between:
- Epistemic Values: Reasons related to the truth or falsity of a scientific claim (e.g., evidence for theory development/testing, predictive accuracy).
- Nonepistemic Values: Reasons not directly related to truth but influencing scientific decisions (e.g., mitigating harm, public credibility, transparency, communication to lay audiences).
- Cognitive Attitudes: How scientists view the scientific literature.
- Book of Truths: Views the literature as a collection of facts, aiming for definitive, replicable results.
- Book of Conversations: Views the literature as a communication tool for exchanging ideas, acknowledging inherent uncertainty and iterative progress.
The framework suggests that replication decisions (whether to replicate and how) are outcomes of evaluating these values through the lens of these cognitive attitudes.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Replication is not a monolithic activity: The "how" and "why" of replication depend heavily on the underlying values and the scientist's perspective.
- The "Book of Truths" vs. "Book of Conversations" dichotomy is powerful: This framing highlights how different goals (e.g., establishing a definitive fact vs. exploring a phenomenon) lead to different replication strategies and interpretations.
- Nonepistemic values are critical drivers: Mitigating harm (e.g., in vaccine trials) or ensuring public trust can strongly influence the need for and rigor of replications, often aligning with a "Book of Truths" approach.
- Transparency is a shared value: While important for both attitudes, it serves different purposes – enabling direct replication for "Book of Truths" and facilitating scientific discourse for "Book of Conversations."
- Sound replication decisions require alignment: A decision to replicate is considered "sound" when it aligns with the researcher's values and cognitive attitude. For example, a large-scale, preregistered trial for a vaccine (mitigating harm, Book of Truths) is a sound decision.
- The scientific literature is a conversation, not just a fact repository: The "Book of Conversations" perspective acknowledges that science is iterative and that not all findings need to be definitive truths. This can lead to more diverse research approaches.
Key Results
The paper doesn't present empirical results but rather a conceptual framework. The "key results" are the insights derived from this framework:
- A structured way for researchers to think about and justify their replication decisions.
- Identification of how epistemic and nonepistemic values, combined with cognitive attitudes, influence the choice of replication tools (e.g., preregistration, large-scale collaborations).
- A proposed set of four questions researchers should ask themselves when planning and reporting replication studies to ensure alignment between their decisions, values, and attitudes.
Limitations and Future Work
- Oversimplification of Cognitive Attitudes: The authors acknowledge that the "Book of Truths" and "Book of Conversations" are caricatures and that real scientists likely hold nuanced positions.
- Defining "Success": The paper touches on the difficulty of defining replication success, noting that statistical methodology alone is insufficient and scientific considerations are equally important. This remains an active area of research.
- Practical Implementation: While the framework is proposed, its widespread adoption and practical implementation in research labs and journals will require further effort.
- Future Work: The authors suggest that this framework can guide future discussions on journal reform, the use of replication tools, and the development of more nuanced approaches to assessing scientific evidence. They also highlight the need for further research into how values and attitudes interact with replication tools.
Summary updated