Introduction to the Special Section on Missing Data

Julie Josse, Jerome P. ReiterStatistical 2018 · DOI

Abstract

Missing data is no longer confined to the realm of survey methodology; it has become a fundamental challenge in the analysis of "found data," including electronic medical records (EMRs) and fused administrative databases. This article introduces a special section of Statistical Science that explores the intersection of classical statistical theory and modern computational techniques. By examining advancements in multiple imputation, double robust estimation, and matrix completion, this section provides a roadmap for addressing the complexities of non-random missingness and high-dimensional data in contemporary research.

The Evolution of the Missing Data Problem

Historically, missing data was primarily viewed through the lens of designed surveys. However, the modern data landscape is dominated by "found data"—information harvested from administrative files, EMRs, and digital traces. In these contexts, missingness is rarely a random occurrence to be "fixed" by simple deletion.

Despite decades of methodological advancement, the most common industry practice remains "complete case analysis" (the removal of any row containing a missing value). This approach is increasingly untenable; not only is it statistically inefficient, but it also introduces significant bias when the data is not Missing Completely at Random (MCAR). As we move toward high-dimensional datasets, the probability of a case being "complete" diminishes, making the development of robust imputation and modeling techniques a necessity rather than a luxury.

Methodological Frontiers

This special section highlights several key areas where statistical methodology is evolving to meet the demands of modern data:

1. Multiple Imputation (MI) and Bayesian Nonparametrics

Moving beyond simple mean-filling, Multiple Imputation accounts for the uncertainty inherent in missing values by generating multiple plausible datasets. Current research, such as that by Murray (2018), advocates for the use of Bayesian nonparametric mixture models (e.g., Dirichlet process mixtures). These models are particularly adept at capturing complex, non-linear relationships and high-dimensional dependencies that traditional linear imputation models fail to represent.

2. Double Robust Estimation

Double robust estimators provide a critical "safety net" for researchers. These methods remain consistent if either the model for the missingness mechanism (the "propensity" of data to be missing) or the model for the outcomes (the data itself) is correctly specified. This redundancy is vital in complex observational studies where the true underlying mechanism is rarely known with certainty.

3. Modeling Missing Not At Random (MNAR) Scenarios

The "nightmare scenario" in missing data occurs when the probability of a value being missing is directly related to the value itself—such as high-income individuals declining to report their earnings. The papers in this section discuss the use of identifying restrictions and sensitivity analysis within Bayesian frameworks to bound the potential bias introduced by MNAR data.

4. Matrix Completion and the ML-Stats Convergence

Borrowing from machine learning, matrix completion techniques use low-rank assumptions to fill in large-scale gaps in data. A recurring theme in this section is the need to integrate these high-scale optimization techniques with statistical paradigms that prioritize uncertainty quantification and the modeling of missingness mechanisms.

Conceptual Insights: Causal Inference and Potential Outcomes

One of the most profound conceptual links established in this collection is the explicit connection between missing data and the Potential Outcomes Framework in causal inference. In a clinical trial or observational study, we seek to understand the effect of a treatment by comparing what would happen to a subject under both Treatment A and Treatment B. Because we can only ever observe one of these states, the unobserved state is, by definition, missing data.

By framing causal inference as a missing data problem, researchers can apply rigorous imputation and weighting techniques—such as the covariate balancing methods demonstrated by Ding and Li (2018)—to improve the validity of causal claims.

Current Research Highlights

The featured articles provide specific guidance for practitioners:

  • Murray (2018): Establishes Bayesian nonparametric mixture models as a gold standard for high-dimensional datasets.
  • Audigier et al. (2018): Offers a practical guide for handling "sporadically" versus "systematically" missing values within multilevel and hierarchical data structures.
  • Ding and Li (2018): Demonstrates how theoretical developments in causal inference can be directly adapted to improve the accuracy of missing data imputation.

Limitations and Future Directions

While the field has progressed significantly, several challenges remain on the horizon:

  • Messy Covariates: Most current research assumes that while outcomes may be missing, covariates are fully observed. Real-world data is rarely this clean.
  • Mixed Mechanisms: There is a lack of robust tools for datasets where different variables exhibit different missingness mechanisms (e.g., some MAR and others MNAR) simultaneously.
  • Multimodal Data: Traditional methodologies are not yet fully equipped for large-scale data involving audio, geospatial, or video components.
  • Black-Box Integration: Future work must investigate how deep learning architectures, such as autoencoders and Generative Adversarial Networks (GANs), can be adapted to provide "statistically proper" imputations that maintain the integrity of downstream scientific inference.

Summary updated

BibTeX
@inproceedings{josse2018introduction,
  author = {Julie Josse and Jerome P. Reiter},
  title = {{Introduction to the Special Section on Missing Data}},
  booktitle = {Statistical},
  year = {2018},
  doi = {10.1214/18-sts332in},
}
Figure