Abstract: Like other approaches to rendering explicit and sampling systematically from a design space of empirical results, integrative experiment design (Almaatouq et al., 2022) can improve the status quo in empirical social and behavioral science. However, reform proposals that start from data generated by sampling and expect to get to good theory misconstrue the role of theory in learning from experiments. Attempts to debias data-driven inferences by “zooming out” are also challenged by the inseparability of the results sampled from a design space from the assumptions that produce them and the difficulty of rendering dependencies explicit without sacrificing interpretability.
Introduction: The Theory Crisis in Social Science
Social and behavioral sciences are currently grappling with a profound "theory crisis." For decades, researchers have often equated "theory" with the simple prediction of an association's direction—for instance, asserting that "increasing variable X will lead to an increase in variable Y." This reductionist approach has contributed to what is known as the "illusion of explanatory power," where models that appear to explain a specific dataset with high precision fail to predict new data or generalize across different contexts.
In response to these failures, a new wave of scientific reform suggests that researchers should "zoom out." This approach, often formalized as Integrative Experiment Design (IED), proposes that instead of focusing on single, isolated experiments, scientists should systematically sample from a massive "design space" of possible experimental conditions and variables. While this method offers a way to map effect heterogeneity, Jessica Hullman’s critique suggests that "zooming out" is not a silver bullet for building robust scientific theories.
The Limits of Integrative Experiment Design
Integrative Experiment Design (IED) and related methods, such as multiverse analysis, aim to provide a more comprehensive view of a phenomenon by running every possible version of an analysis or sampling across diverse conditions. The goal is to move beyond the "status quo" of narrow, brittle findings. However, Hullman identifies several philosophical and methodological pitfalls in assuming that more data naturally leads to better theory.
1. Theory as Mechanism, Not Trend
A fundamental insight of Hullman’s critique is that a genuine theory must propose a data-generating mechanism rather than just identifying a trend. While sampling a vast design space can improve predictive accuracy, it does not automatically reveal the "why" behind the observations. A model can guess an outcome correctly without explaining the underlying causal process. Without a formal mechanism, researchers risk falling into the trap of Meehl’s "Crud Factor"—the observation that in social sciences, nearly everything is correlated to some degree. If one looks at enough variables, "significant" results are inevitable, but they may be theoretically vacuous.
2. The Myth of Unconditional Data
A central promise of "zooming out" is the potential to debias inferences by looking at the bigger picture. However, Hullman argues that there is no such thing as "raw" or "unconditional" data. Even when a researcher samples from a large design space, the resulting data remains inseparable from the assumptions made when defining that space. The inferences drawn are always conditional on the initial parameters, meaning that sheer volume cannot "debias" a study if the underlying conceptual framework is flawed.
3. The Goldilocks Problem and Interpretability
IED assumes the existence of a "Goldilocks" zone of dimensionality—a model complex enough to capture reality but simple enough for human researchers to understand. Hullman warns of an "interpretability trap": as we increase the complexity of our designs to capture more heterogeneity, we risk creating "surrogate models" that are as opaque as the phenomena they aim to explain. Without a formal "theory of interpretability," there is no guarantee that these large-scale integrative models will be useful for human scientific reasoning.
Conclusion: Zooming Out and Zooming In
Hullman concludes that while integrative modeling is a significant improvement over isolated, narrow experiments, it cannot replace the need for formal theory. The "gesture of completeness" provided by large-scale data sampling can be rhetorically persuasive but scientifically misleading if the theoretical foundation remains weak.
To truly reform social science, Hullman suggests a dual approach:
- Zoom Out: Use integrative designs to understand how effects vary across different contexts and to debunk the illusion of robustness in narrow studies.
- Zoom In: Use formal frameworks to define exactly what is expected to happen within a specific task and why, focusing on the data-generating mechanism.
Ultimately, data-driven inferences cannot substitute for the hard work of theoretical formalization. Future progress in scientific reform must involve not only better sampling of the design space but also a more transparent accounting of the assumptions and dependencies that define our learning problems.
Summary updated