Some problems with zooming out as scientific reform

Jessica Hullman2023 · DOI

Abstract

Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al. 2023) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments. Attempts to debias data-driven inferences by “zooming out” are also challenged by the inseparability of the results sampled from a design space from the assumptions that produce them, and the difficulty of rendering dependencies explicit without sacrificing interpretability.

Introduction

The social and behavioral sciences are currently grappling with a "theory crisis." For decades, the standard operating procedure has involved conducting narrow experiments where "theory" is reduced to predicting the mere direction of an association—for instance, asserting that "X will increase Y." This reductionist approach fails on two fronts: it provides weak experimental tests that are easily passed, and it fails to explain why results fluctuate wildly across different contexts and populations.

In response, reformers have proposed "integrative experiment design." This framework suggests that researchers should "zoom out" by defining a massive "design space" of possible experimental conditions and sampling systematically from it. The goal is to move beyond isolated findings toward a more generalizable understanding of human behavior. However, as Jessica Hullman argues, this strategy contains hidden pitfalls that may prevent it from being the panacea the field hopes for.

The Logic and Limits of "Zooming Out"

Integrative experiment design relies on two primary methodological trends:

  1. Integrative Modeling: Sampling from a high-dimensional space of experimental variables to construct "surrogate models" that predict outcomes across various conditions.
  2. Multiverse Analysis: Executing all plausible analytical paths for a single dataset to determine how sensitive a result is to specific researcher choices.

While these methods are a significant improvement over the status quo—which often ignores effect heterogeneity entirely—they remain tethered to the subjective assumptions made when the design space was first defined. Hullman posits that these data-driven approaches cannot automatically generate "unbiased" or "unconditional" truths because the boundaries of the search space are themselves a product of researcher bias.

The "Crud Factor" and the Theory Gap

A central challenge to the "zooming out" strategy is what psychologist Paul Meehl termed the "crud factor." In the social sciences, nearly everything is weakly correlated with everything else. If researchers zoom out too far without a theoretical compass, they encounter a sea of potential influences—ranging from a participant's religious orientation to the morning's news headlines.

Without a formal theory to constrain the search, researchers risk "drowning in complexity." Hullman argues that theory should be viewed as a data-generating mechanism rather than a map. One cannot simply sample their way into a robust theory; rather, a formal model of how the world works is required to know which variables are worth testing and which are merely part of the "crud."

The Interpretability Paradox

Integrative designs aim for a "Goldilocks" zone of complexity: models that are more sophisticated than a single-variable experiment but simple enough for a human to interpret. However, Hullman points out an inherent paradox: there is currently no formal "theory of interpretability" to guide these choices.

When we choose to include certain variables and exclude others to keep a model interpretable, we are making value judgments. Consequently, any "unbiased" evaluation produced by zooming out is still conditional on the arbitrary boundaries of the design space. The data cannot be separated from the assumptions used to generate the learning problem in the first place.

Conclusion: Balancing the Lens

The move toward integrative experiment design is a step in the right direction, particularly in its acknowledgment of how effects vary across contexts. However, it is not a complete solution for scientific reform.

Hullman concludes that "zooming out" (sampling more broadly) must be balanced with "zooming in"—the rigorous work of formalizing expectations about specific tasks and being explicit about the conditions under which a claim is expected to hold. The rhetorical appeal of "completeness" offered by large-scale sampling can be misleading if it isn't grounded in a mechanistic understanding of the subject matter. Ultimately, scientific reform should focus not just on gathering more data or building bigger design spaces, but on asking better-defined, theoretically grounded questions.

Summary updated

Abstract

Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al. 2023) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments. Attempts to debias data-driven inferences by “zooming out” are also challenged by the inseparability of the results sampled from a design space from the assumptions that produce them, and the difficulty of rendering dependencies explicit without sacrificing interpretability.
BibTeX
@misc{hullman2023some,
  author = {Jessica Hullman},
  title = {{Some problems with zooming out as scientific reform}},
  year = {2023},
  doi = {10.31222/osf.io/bg36w},
}
Figure