Raiders of the lost HARK: a reproducible inference framework for big data science

Mattia Prosperi, Jiang Bian, Iain Buchan, James S. Koopman, Matthew Sperrin, Mo WangPalgrave 2019 · DOI

Abstract

Hypothesizing after the results are known (HARK) has been disparaged as data dredging, and safeguards including hypothesis preregistration and statistically rigorous oversight have been recommended. Despite potential drawbacks, HARK has deepened thinking about complex causal processes. Some of the HARK precautions can conflict with the modern reality of researchers’ obligations to use big, ‘organic’ data sources—from high-throughput genomics to social media streams. We here propose a HARK-solid, reproducible inference framework suitable for big data, based on models that represent formalization of hypotheses. Reproducibility is attained by employing two levels of model validation: internal (relative to data collated around hypotheses) and external (independent to the hypotheses used to generate data or to the data used to generate hypotheses). With a model-centered paradigm, the reproducibility focus changes from the ability of others to reproduce both data and specific inferences from a study to the ability to evaluate models as representation of reality. Validation underpins ‘natural selection’ in a knowledge base maintained by the scientific community. The community itself is thereby supported to be more productive in generating and critically evaluating theories that integrate wider, complex systems.

Introduction: The HARK Dilemma in Big Data

In classical scientific inquiry, the hypothetico-deductive method serves as the gold standard: researchers formulate a hypothesis, design an experiment, and then collect data to test that hypothesis. Deviating from this path by "Hypothesizing After the Results are Known" (HARK) is traditionally disparaged as "data dredging" or "p-hacking." Such practices are often blamed for the reproducibility crisis, as they can lead to false positives that fail to hold up in independent studies.

However, the rise of "big, organic data"—massive, pre-existing datasets like genomic sequences, social media streams, or sensor networks—challenges this traditional paradigm. In these contexts, the data already exists, and the number of potential analysis paths is virtually infinite. Strict preregistration of a single hypothesis can be counterproductive, stifling the exploratory discovery necessary to understand complex systems. The central challenge is how to enable rigorous, reproducible science in an era where exploratory discovery is not just common, but often necessary.

Methodology: Shifting to a Model-Centric Approach

The authors propose moving from a "data-centric" view of science to a "model-centric" one. This shift involves formalizing hypotheses as computable models—such as functions, probability distributions, or logical assertions—that can be empirically falsified.

A key component of this methodology is the adoption of Hybrid Reasoning. Traditional science often oscillates between deduction (testing theories) and abduction (finding the most likely explanation for observed data). The proposed framework integrates these into a continuous cycle.

Figure 4: The integration of abduction and deduction into a hybrid abductive-deductive reasoning cycle.

By treating hypotheses as formal models, the focus of reproducibility shifts. It is no longer just about whether another researcher can run the same code on the same data to get the same p-value, but whether the proposed model accurately represents reality when applied to new, independent data.

The HARK-Solid Inference Framework

The core of the proposal is the HARK-solid inference framework. This framework decouples the generation of a hypothesis from its validation. In this view, it does not matter how a hypothesis was found—whether through a flash of insight or exhaustive data mining—as long as it survives a rigorous, two-level validation process.

  1. Internal Validation: Using techniques like cross-validation or bootstrapping on the initial dataset to ensure the model captures genuine patterns rather than random noise.
  2. External Validation: The "gold standard" of the framework. This involves testing the model against entirely independent datasets or in different contexts to verify its generalizability.
Figure 5: The HARK-solid knowledge inference roadmap, illustrating the interplay between hypotheses, data, and multi-level validation.

As shown in the roadmap above, scientific progress is viewed as a "natural selection" of models. Validation acts as the selective pressure, where models that fail to generalize are discarded or refined, while "fit" models are integrated into the community's collective knowledge base.

Case Studies: Complexity in Practice

The paper demonstrates the necessity of this framework through three complex domains:

  • Phylogenetics: Reconstructing evolutionary trees involves navigating an unmanageably large space of possible "paths," making traditional experimental design impossible.
  • Genomics (GWAS): Modern "multi-omics" research involves millions of interacting variables, requiring exploratory models to uncover non-linear relationships.
  • Social Media Analysis: The volatility and inherent sampling biases of platforms like Twitter make traditional fixed-hypothesis testing inadequate for capturing shifting social dynamics.

Key Insights: HARK as a Virtue

The authors argue that when dealing with complex systems, HARKing should be viewed not as a sin, but as a virtue that facilitates the generation of theories integrating wider systems. By making the process transparent and emphasizing external validation, "data dredging" is transformed into a robust discovery mechanism. This approach prioritizes effect sizes and "goodness of fit" over the often-misleading binary of p-values.

Limitations and Future Directions

While the HARK-solid framework offers a path forward, it faces practical hurdles. Finding high-quality, truly independent datasets for external validation is often difficult and costly. Furthermore, there is no universal metric for "goodness of fit"; validation criteria must be tailored to specific fields, from epidemiology to social science. Future research should focus on empirically tracking how external validation affects the "survival" of models in scientific literature to see if this "natural selection" process is functioning as intended.

Summary updated

Abstract

Abstract Hypothesizing after the results are known (HARK) has been disparaged as data dredging, and safeguards including hypothesis preregistration and statistically rigorous oversight have been recommended. Despite potential drawbacks, HARK has deepened thinking about complex causal processes. Some of the HARK precautions can conflict with the modern reality of researchers’ obligations to use big, ‘organic’ data sources—from high-throughput genomics to social media streams. We here propose a HARK-solid, reproducible inference framework suitable for big data, based on models that represent formalization of hypotheses. Reproducibility is attained by employing two levels of model validation: internal (relative to data collated around hypotheses) and external (independent to the hypotheses used to generate data or to the data used to generate hypotheses). With a model-centered paradigm, the reproducibility focus changes from the ability of others to reproduce both data and specific inferences from a study to the ability to evaluate models as representation of reality. Validation underpins ‘natural selection’ in a knowledge base maintained by the scientific community. The community itself is thereby supported to be more productive in generating and critically evaluating theories that integrate wider, complex systems.
BibTeX
@inproceedings{prosperi2019raiders,
  author = {Mattia Prosperi and Jiang Bian and Iain Buchan and James S. Koopman and Matthew Sperrin and Mo Wang},
  title = {{Raiders of the lost HARK: a reproducible inference framework for big data science}},
  booktitle = {Palgrave},
  year = {2019},
  doi = {10.1057/s41599-019-0340-8},
}
Figure