Supplemental Material for Creative or Not? Hierarchical Diffusion Modeling of the Creative Evaluation Process

Michelle C. Donzallaz, Julia M. Haaf, Claire E. StevensonJournal of Experimental Psychology Learning Memory and Cognition 2022 · DOI

TL;DR

This paper details the open science practices employed in a study investigating the creative evaluation process. The authors provide public access to their data, materials, and preregistered analysis plan, demonstrating a commitment to transparency and reproducibility in psychological research.

What Problem Does This Paper Address?

The paper addresses the broader challenge within scientific research of ensuring transparency, reproducibility, and openness. Specifically, it details how the authors of the study "Creative or Not? Hierarchical Diffusion Modeling of the Creative Evaluation Process" adhered to open science principles by making their research artifacts publicly available. This practice aims to combat issues like p-hacking, publication bias, and the "replication crisis" by allowing other researchers to scrutinize, verify, and build upon their work. The context here is the growing movement in psychology and other fields to adopt practices like preregistration and data sharing to enhance the rigor and trustworthiness of research findings.

Methods

The paper itself is not a research paper detailing novel methods in creative evaluation. Instead, it is a disclosure document outlining the open science practices adopted by the authors of the aforementioned study. The "methods" here refer to the adherence to open science principles:

  • Open Data Badge: The authors have made their dataset publicly available via the Open Science Framework (OSF).
  • Open Materials Badge: The authors have made their research materials publicly available via OSF.
  • Preregistered Badge: The authors preregistered their analysis plan prior to examining the data.

The document specifies the URLs for these repositories and confirms that the information provided is sufficient for independent reproduction of results and methodology.

Datasets

The paper refers to a dataset collected for the study "Creative or Not? Hierarchical Diffusion Modeling of the Creative Evaluation Process." This dataset is made publicly available via the Open Science Framework (OSF) at the provided URL: https://osf.io/73c2d/.

The "Alternative note" section provides specific details about the data collection for Study 2 within the larger research:

  • Initially, the preregistration planned to include only data from Dutch native speakers.
  • Retrospectively, the decision was made to include participants who could read and respond fluently in Dutch, regardless of their native language, if they chose the Dutch version of the experiment.
  • Participants were required to read instructions in Dutch to complete the experiment in Dutch, and fluency was assessed based on their ability to read and write a few simple phrases in Dutch.

Key Insights and Ideas

  • Commitment to Transparency: The authors' proactive disclosure of their data, materials, and preregistered plan is a significant insight into the practical application of open science principles. This goes beyond simply conducting research to actively facilitating its verification and extension.
  • Preregistration as a Safeguard: The preregistration of the analysis plan before data examination is highlighted as a key step to prevent confirmation bias and ensure that the reported findings are not the result of post-hoc data dredging.
  • Flexibility and Justification in Open Science: The "Alternative note" demonstrates that open science doesn't necessarily mean rigid adherence to an initial plan. It allows for justified deviations, provided these changes are transparently documented and explained, as indicated by the "TC" (Transparent Changes) notation. This acknowledges the iterative nature of research while maintaining rigor.
  • Inclusivity in Data Collection: The decision to include non-native Dutch speakers who demonstrated fluency highlights a thoughtful approach to expanding the generalizability of findings while still maintaining methodological integrity.

Key Results

This document does not present quantitative results from the study itself. Instead, it confirms the successful implementation of open science practices:

  • Open Data: The data is publicly available at https://osf.io/73c2d/ and is deemed sufficient for reproducing all reported results.
  • Open Materials: The materials are publicly available at https://osf.io/73c2d/ and are deemed sufficient for reproducing the reported methodology.
  • Preregistered Plan: The analysis plan was preregistered at https://osf.io/73c2d/ prior to data examination.
  • Transparent Changes: There were changes made to the preregistered analysis plan for the primary confirmatory analysis, which are explained in the "Alternative note." This will result in the "TC" notation being added to the badge.
  • Completeness of Reporting: All analyses described in the registered plan are reported in the article.

Limitations and Future Work

  • Third-Party Material Permissions: The authors affirm that any third-party material has been reproduced or made available with proper permission. This highlights a potential challenge in open science: ensuring all components of a study are cleared for public sharing.
  • Confidentiality: The "Alternative note" implicitly acknowledges that sharing participant data can raise confidentiality concerns, which is why a mechanism for explaining such circumstances is provided.
  • Future Work (Implied): The document's existence and the detailed disclosure suggest a commitment to ongoing open science practices in future research by these authors. The "Alternative note" also points to the need for researchers to be prepared to justify deviations from preregistered plans, which is a valuable aspect of transparent research.

Summary updated

Abstract

departmental websites do not qualify as repositories.
BibTeX
@article{donzallaz2022supplemental,
  author = {Michelle C. Donzallaz and Julia M. Haaf and Claire E. Stevenson},
  title = {{Supplemental Material for Creative or Not? Hierarchical Diffusion Modeling of the Creative Evaluation Process}},
  journal = {Journal of Experimental Psychology Learning Memory and Cognition},
  year = {2022},
  doi = {10.1037/xlm0001177.supp},
}
Figure