Moderated Online Data-Collection for Developmental Research: Methods and Replications

Aaron Chuey, Mika Asaba, Sophie Bridgers, Brandon Allen Carrillo, Griffin Dietz, Teresa García, Julia Leonard, Shari Liu, Megan Merrick, Samaher Radwan, Jessa Stegall, Natalia Vélez, Brandon Matthew Woo, Yang Wu, Xi J. Zhou, Michael C. Frank, Hyowon GweonFrontiers in Psychology 2021 · DOI

TL;DR

This paper validates the use of moderated online data collection methods for developmental research, demonstrating that results from online studies using video conferencing are comparable to traditional in-person studies. This opens doors for more accessible, diverse, and large-scale developmental research.

What Problem Does This Paper Address?

The COVID-19 pandemic accelerated the adoption of online data collection in developmental psychology. However, many researchers remain hesitant due to uncertainties about the validity of these methods, particularly for young children and infants. Specifically, there's a lack of clear guidance on how to effectively design and implement online studies, and limited direct comparisons between in-person and online data collection for the same tasks. This paper aims to bridge this gap by providing practical advice and empirical evidence for the reliability of moderated online data collection.

Methods

The authors propose and evaluate "moderated online data collection," which involves real-time, face-to-face interactions with participants via video conferencing software (e.g., Zoom). This contrasts with "unmoderated" online methods where participants complete tasks independently. The paper outlines best practices for designing these moderated sessions, focusing on maintaining participant engagement and addressing potential technical issues.

To validate these methods, the authors present four studies that were conducted both in-person and via moderated online sessions. These studies employed different developmental measures:

  • Choice and Verbal Measures (Studies 1 & 2): These studies involved children aged 4-9 making choices or providing verbal responses, adapted for online by using visual cues and verbal prompts.
  • Looking Time Measures (Studies 3 & 4): These studies focused on infants, using preferential looking and violation-of-expectation paradigms, adapted for online by presenting stimuli via screen-sharing and measuring gaze duration.

A key methodological contribution is the direct comparison of results from the in-person and online versions of each study, followed by a meta-analysis to aggregate findings across all four studies.

Datasets

Not explicitly discussed as distinct datasets. The paper presents data from four newly conducted studies (Studies 1-4) that were run in parallel in-person and online. The data from Studies 1, 2, and 4 are available via OSF, and data for Study 3 is referenced as being in Woo and Spelke (2020).

Key Insights and Ideas

  • Moderated online studies are a viable alternative: The core finding is that, for the measures tested, moderated online data collection yields results comparable to traditional in-person methods. This is a significant finding for the field.
  • Adaptability of measures: The paper demonstrates that common developmental measures, including choice tasks, verbal responses, and infant looking-time paradigms, can be successfully adapted for moderated online settings.
  • Importance of careful design: While online methods offer flexibility, careful consideration of factors like software choice, participant engagement strategies (e.g., keeping sessions short, using social cues), and robust handling of technical issues is crucial for successful implementation.
  • Potential for increased accessibility and diversity: Online methods have the potential to overcome geographical barriers, allowing researchers to recruit from more diverse populations and conduct larger-scale studies, which is vital for the generalizability of findings.

Key Results

The four studies presented showed consistent patterns between in-person and online data collection:

  • Study 1 (4-5 year olds, competence inference): Children's performance in inferring agent competence based on task difficulty was comparable across in-person (90% correct) and online (85% correct) conditions.
  • Study 2 (6-9 year olds, social cues): Children's ability to use adult surprise to infer competence was also similar, with online results mirroring those of the older age group in the in-person study.
  • Study 3 (15-month-olds, social evaluations): Infants' preferential reaching (in-person) and looking (online) patterns regarding agents' intentions were comparable.
  • Study 4 (6-month-olds, cost minimization): Infants' looking times at efficient versus inefficient actions were similar across in-person and online conditions.

The meta-analysis of these studies estimated a small, non-significant negative effect of online data collection (p = 0.58), indicating no substantial difference in effect sizes between the two methods.

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on social cognition: The studies primarily involved animated agents, which may inherently be more engaging for children. The generalizability of these findings to studies involving inanimate objects, sounds, or more abstract stimuli remains an open question.
  • Limited range of measures: The paper focused on choice, verbal, and looking-time measures. The efficacy of other, more continuous measures (e.g., rating scales, free-form responses) in moderated online settings needs further investigation.
  • Participant demographics: The studies were conducted in the US with participants likely having reliable internet access and familiarity with technology. The applicability to populations with limited internet access or lower digital literacy is not yet established.

Future work should explore a broader range of measures and research questions, investigate the efficacy of online methods in diverse populations, and continue to develop innovative strategies for online data collection. The authors express optimism that online methods will continue to improve the accessibility, diversity, and replicability of developmental science.

Summary updated

Abstract

Online data collection methods are expanding the ease and access of developmental research for researchers and participants alike. While its popularity among developmental scientists has soared during the COVID-19 pandemic, its potential goes beyond just a means for safe, socially distanced data collection. In particular, advances in video conferencing software has enabled researchers to engage in face-to-face interactions with participants from nearly any location at any time. Due to the novelty of these methods, however, many researchers still remain uncertain about the differences in available approaches as well as the validity of online methods more broadly. In this article, we aim to address both issues with a focus on moderated (synchronous) data collected using video-conferencing software (e.g., Zoom). First, we review existing approaches for designing and executing moderated online studies with young children. We also present concrete examples of studies that implemented choice and verbal measures (Studies 1 and 2) and looking time (Studies 3 and 4) across both in-person and online moderated data collection methods. Direct comparison of the two methods within each study as well as a meta-analysis of all studies suggest that the results from the two methods are comparable, providing empirical support for the validity of moderated online data collection. Finally, we discuss current limitations of online data collection and possible solutions, as well as its potential to increase the accessibility, diversity, and replicability of developmental science.
BibTeX
@inproceedings{chuey2021moderated,
  author = {Aaron Chuey and Mika Asaba and Sophie Bridgers and Brandon Allen Carrillo and Griffin Dietz and Teresa García and Julia Leonard and Shari Liu and Megan Merrick and Samaher Radwan and Jessa Stegall and Natalia Vélez and Brandon Matthew Woo and Yang Wu and Xi J. Zhou and Michael C. Frank and Hyowon Gweon},
  title = {{Moderated Online Data-Collection for Developmental Research: Methods and Replications}},
  booktitle = {Frontiers in Psychology},
  year = {2021},
  doi = {10.3389/fpsyg.2021.734398},
}

Loading related content...

Figure