Scaling up experimental social, behavioral, and economic science

Abdullah Almaatouq, Joshua Becker, Michael S. Bernstein, Robert I. Botto, Eric T. Bradlow, Ekaterina Damer, Angela Duckworth, Thomas L. Griffiths, Joshua K. Hartshorne, Edith Law, David Lazer, Min Liu, J. Nathan Matias, David G. Rand, Matthew Salganik, Emma Satlof‐Bedrick, Maurice E. Schweitzer, Hirokazu Shirado, Jordan W. Suchow, Siddharth Suri, Milena Tsvetkova, Duncan J. Watts, Mark E. Whiting, Ming Yin2021 · DOI

The Computational Social Science Lab at the University of Pennsylvania.

Abstract

The standard experimental paradigm in the social, behavioral, and economic (SBE) sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not generalize easily to real-world contexts or even to other variations of the same experiment. As a result, there exist few universally acknowledged findings, and even those are occasionally overturned by new data. We argue that to achieve replicable, generalizable, scalable and ultimately useful social and behavioral science, a fundamental rethinking of the model of virtual-laboratory style experiments is required. Not only is it possible to design and run experiments that are radically different in scale and scope than was possible in an era of physical labs; this ability allows us to ask fundamentally different types of questions than have been asked historically of lab studies. We posit, however, that taking full advantage of this new and exciting potential will require four major changes to the infrastructure, methodology, and culture of experimental science: (1) significant investments in software design and participant recruitment, (2) innovations in experimental design and analysis of experimental data, (3) adoption of new models of collaboration, and (4) a new understanding of the nature and role of theory in experimental social and behavioral science.

Introduction: The Generalizability Crisis

For decades, the gold standard of SBE research has been the laboratory experiment. While powerful, these studies suffer from three major bottlenecks that limit their real-world utility. First is the "WEIRD" problem: participants are overwhelmingly from Western, Educated, Industrialized, Rich, and Democratic societies, making them unrepresentative of the global population. Second is the complexity gap; real-world social phenomena involve large populations interacting over months or years, whereas lab studies usually involve small groups interacting for minutes in highly simplified settings. Finally, the fragility of findings stems from the high cost of experiments, which forces researchers to test only a tiny slice of a problem. This leads to results that often fail to hold up when parameters change even slightly.

While "virtual labs" (online experiments) have emerged, they are currently being used primarily to replicate the limitations of physical labs rather than transcending them. To move forward, we must move beyond simply putting traditional experiments online and instead build a high-throughput, modular infrastructure capable of testing complex, multi-factor theories at the scale of entire societies.

A Roadmap for High-Throughput Social Science

To achieve this vision, we propose a fundamental shift in research methodology focused on four key pillars:

  • High-Throughput Infrastructure: We must move away from "one-off" custom code toward a modular ecosystem of interoperable software components. This approach abstracts away the technical overhead, much like cloud computing did for server management.
  • Procedural Design: Instead of hand-picking a few conditions to test, researchers should use algorithms to generate and sample thousands of variations across a massive parameter space.
  • Adaptive & Algorithmic Designs: Techniques like "iterated learning"—where one participant's output becomes the next's input—can be used to implement complex algorithms using human participants as the processing units.
  • Mass Collaboration: The culture must shift from "lone wolf" investigators to large-scale, multi-lab collaborations (such as "Many Labs" projects) to handle the administrative and ethical overhead of massive studies.

Theoretical Shifts and Mapping Social Levers

Visualizing the complex interconnections of social and behavioral data. Source, p. 3

Visualizing the complex interconnections of social and behavioral data.

Historically, SBE scientists preferred simple theories because data was scarce. However, as we scale up, we must embrace the bias-variance trade-off. Simple models are often too biased to capture reality; we must move toward more complex models that can actually predict behavior in the wild. In group dynamics, for example, there are dozens of "levers"—such as diversity, task complexity, and communication rules—that affect performance. Instead of asking binary questions like "Does diversity help?", we should be mapping the entire multi-dimensional surface to see when and how these factors interact.

This scaling up has already shown promise in resolving long-standing academic debates. For instance, by using an online study with 4,000 participants (compared to original small lab samples), researchers were able to map the full performance-to-self-assessment curve of the Dunning-Kruger effect, providing the resolution needed to identify asymmetries that smaller studies missed. Furthermore, "macro" experiments are now proving that the unit of analysis can be an entire market or community, making societal-scale experimentation technically feasible today.

Overcoming Cultural and Administrative Barriers

The primary obstacles to this new era are not technological, but cultural and administrative. Generations of researchers have internalized the constraints of the physical lab as "sound science," and breaking the habit of testing one hypothesis at a time is a major hurdle. Furthermore, current recruitment platforms like Amazon Mechanical Turk were built for simple labeling, not complex, synchronous social interaction. We need new platforms that support long-term panels and simultaneous recruitment of thousands of people.

Finally, academic incentive structures must change to reward the massive, multi-author collaborations required for this vision. While the path is ambitious, the financial and technical investments required are dwarfed by the potential for a new era of scientific advance in social, behavioral, and economic studies.

Summary updated

Abstract

The standard experimental paradigm in the social, behavioral, and economic sciences is extremely limited. Although recent advances in digital technologies and crowdsourcing services allow individual experiments to be deployed and run faster than in traditional physical labs, a majority of experiments still focus on one-off results that do not generalize easily to real-world contexts or even to other variations of the same experiment. As a result, there exist few universally acknowledged findings, and even those are occasionally overturned by new data. We argue that to achieve replicable, generalizable, scalable and ultimately useful social and behavioral science, a fundamental rethinking of the model of virtual-laboratory style experiments is required. Not only is it possible to design and run experiments that are radically different in scale and scope than was possible in an era of physical labs; this ability allows us to ask fundamentally different types of questions than have been asked historically of lab studies. We posit, however, that taking full advantage of this new and exciting potential will require four major changes to the infrastructure, methodology, and culture of experimental science: (1) significant investments in software design and participant recruitment, (2) innovations in experimental design and analysis of experimental data, (3) adoption of new models of collaboration, and (4) a new understanding of the nature and role of theory in experimental social and behavioral science. We conclude that the path we outline, although ambitious, is well within the power of current technology and has the potential to facilitate a new class of scientific advances in social, behavioral and economic studies.This paper emerged from discussions at a workshop held by the Computational Social Science Lab at the University of Pennsylvania in January 2020. The work was supported by James and Jane Manzi Analytics Fund and the Alfred P. Sloan Foundation.
BibTeX
@misc{almaatouq2021scaling,
  author = {Abdullah Almaatouq and Joshua Becker and Michael S. Bernstein and Robert I. Botto and Eric T. Bradlow and Ekaterina Damer and Angela Duckworth and Thomas L. Griffiths and Joshua K. Hartshorne and Edith Law and David Lazer and Min Liu and J. Nathan Matias and David G. Rand and Matthew Salganik and Emma Satlof‐Bedrick and Maurice E. Schweitzer and Hirokazu Shirado and Jordan W. Suchow and Siddharth Suri and Milena Tsvetkova and Duncan J. Watts and Mark E. Whiting and Ming Yin},
  title = {{Scaling up experimental social, behavioral, and economic science}},
  year = {2021},
  doi = {10.31219/osf.io/wksv8},
}
Figure