The "Given Data" Paradigm Undermines Both Cultures

Tyler H. McCormickObservational 2021 · DOI

TL;DR

This paper argues that the traditional "black box" view of statistical modeling, focusing solely on the transformation of input data to output, is too narrow. By expanding our perspective to include the crucial stages before data collection and after model output, we can unlock new avenues for statistical innovation and better address real-world problems, especially with imperfect data.

What Problem Does This Paper Address?

The paper revisits Leo Breiman's influential 2001 paper, "Statistical modeling: The two cultures," which described a divide in statistics between "data modelers" (focused on understanding nature) and "algorithmic modelers" (focused on prediction). Breiman framed this distinction around what happens inside a "black box" that transforms input data into output. The author contends that this "given data" paradigm, which treats data as a fixed input, is limiting. It overlooks the significant statistical challenges and opportunities that lie in the processes before data are collected (design, collection, and preparation) and after a model produces results (interpretation, communication, and decision-making). This narrow focus can lead to over-reliance on complex models for extrapolation with limited data and a failure to fully leverage new data sources.

Methods

The paper doesn't introduce new algorithms or formal methods in the traditional sense. Instead, it proposes a conceptual shift in how we view and practice statistics. The author advocates for broadening the scope of statistical inquiry beyond the "black box" of model fitting. This involves:

  • Emphasizing "Before" the Box: Focusing on the design of data collection, understanding the limitations of passively observed data (like administrative records), and investing in data transformation and representation to make data suitable for analysis. This includes areas like survey sampling.
  • Emphasizing "After" the Box: Concentrating on the faithful communication of uncertainty and the quality of information in estimates to stakeholders, particularly policymakers. This involves areas like data visualization and understanding the implications of estimates for decision-making.
  • Integrating Algorithmic and Data Modeling: Recognizing that modern problems often require a combination of both predictive (algorithmic) and descriptive (data modeling) approaches, and that the uncertainty from both needs to be quantified and propagated.

Datasets

Not discussed in the paper. The author uses hypothetical examples from their work in global health, particularly concerning the estimation of cause-of-death distributions in settings with sparse data.

Key Insights and Ideas

  • The "Given Data" Paradigm is a False Partition: Focusing solely on the "black box" creates an artificial separation between methodology development and its application to static data, hindering innovation.
  • "Before" and "After" are Rich Areas for Statistical Practice: The design of data collection and the communication of results are not mere pre- or post-processing steps but are fundamental statistical challenges with significant impact.
  • New Data Sources Require New Statistical Thinking: The proliferation of passively observed data (e.g., electronic health records) necessitates more effort in data transformation and representation, moving beyond simply applying existing models.
  • Uncertainty is Multi-faceted: When combining predictive and descriptive modeling, uncertainty arises not only from sampling distributions but also from the prediction process itself, requiring careful quantification.
  • Statistics Offers a Comparative Advantage: Statisticians are uniquely positioned to lead in addressing the challenges of data collection design and the nuanced communication of uncertainty, which are critical for informed decision-making.

Key Results

The paper presents a qualitative argument rather than quantitative results. The key "results" are conceptual:

  • A call to shift the focus of statistical practice, research, and education towards the stages "before" and "after" the modeling "black box."
  • An argument that embracing these broader perspectives will help statisticians grasp new opportunities and avoid being constrained by familiar tools and problems.
  • The author expresses optimism about the future of statistics, believing its comparative advantage is more salient than ever when viewed through this expanded lens.

Limitations and Future Work

The author acknowledges that their "outside the box" perspective is not entirely unique and that similar themes exist in existing literature. The paper's primary limitation is its conceptual nature; it doesn't present new empirical findings or methodological advancements but rather a framework for thinking about statistical practice.

Future work suggested by the paper includes:

  • Further exploration of how to effectively integrate "before" and "after" considerations into statistical education and research.
  • Developing methods for quantifying and propagating uncertainty that accounts for both predictive and descriptive modeling steps, as well as data collection limitations.
  • Investigating how to best communicate complex statistical findings and their inherent uncertainties to non-expert stakeholders, particularly policymakers.

Summary updated

Abstract

Breiman organizes Statistical modeling: The two cultures around a simple visual. Data, to the far right, are compelled into a "black box" with an arrow and then catapulted left by a second arrow, having been transformed into an output. Breiman then posits two interpretations of this visual as encapsulating a distinction between two cultures in statistics. The divide, he argues is about what happens in the "black box." In this comment, I argue for a broader perspective on statistics and, in doing so, elevate questions from "before" and "after" the box as fruitful areas for statistical innovation and practice.
BibTeX
@inproceedings{mccormick2021given,
  author = {Tyler H. McCormick},
  title = {{The "Given Data" Paradigm Undermines Both Cultures}},
  booktitle = {Observational},
  year = {2021},
  doi = {10.1353/obs.2021.0002},
}

Loading related content...

Figure