What Is a Good Imputation Under MAR Missingness?

Jeffrey Näf, Erwan Scornet, Julie JossearXiv 2024 · arXiv

TL;DR

This paper provides a theoretical foundation for why "Fully Conditional Specification" (FCS)—the logic behind the popular MICE algorithm—is a valid way to handle missing data under the Missing at Random (MAR) assumption. The authors introduce mice-DRF, a new method using Distributional Random Forests, and argue that we must stop using RMSE to evaluate imputations, advocating for distributional metrics like "energy distance" instead.

What Problem Does This Paper Address?

Missing data is a plague in data science. Most researchers rely on the Missing at Random (MAR) assumption, which suggests that the reason data is missing can be fully explained by the variables we did observe. However, the field has been flooded with new imputation algorithms without a clear consensus on what makes one "good."

The authors identify two major gaps:

  1. Theoretical Uncertainty: While "joint modeling" (imputing everything at once) is well-understood, it was less clear if "fully conditional specification" (imputing one variable at a time in a loop) could actually recover the true underlying distribution in a nonparametric setting.
  2. The RMSE Trap: Most benchmarks use Root Mean Squared Error (RMSE) to rank methods. The authors argue this is fundamentally flawed because RMSE rewards methods that guess the "average" value, which narrows the variance and destroys the relationships between variables needed for downstream analysis.

Methods

The paper focuses on the Fully Conditional Specification (FCS) framework. In FCS, you iteratively train a model for each variable $X_j$ using all other variables as predictors.

The authors propose three requirements for an "ideal" imputation method:

  1. Distributional Regression: It should provide a draw from a distribution, not just a single point estimate (like a mean).
  2. Nonlinearity/Interactions: It must capture complex, non-linear relationships.
  3. Robustness to Covariate Shifts: It must handle cases where the distribution of predictors looks different in the "missing" group compared to the "observed" group.

To meet these, they introduce mice-DRF. This method plugs a Distributional Random Forest (DRF) into the MICE (Multiple Imputation by Chained Equations) loop. Unlike standard Random Forests that target the conditional mean, DRF targets the entire conditional distribution, allowing for more realistic "draws" of missing values.

Datasets

The authors use a mix of synthetic and real-world data:

  • Synthetic "Toy" Examples: Seven distinct scenarios designed to test specific mathematical conditions (e.g., different types of MAR, varying levels of overlap, and nonlinear dependencies).
  • Air Quality Data: A real-world dataset from the US Environmental Protection Agency (EPA) containing ~50,000 observations and 11 variables. They artificially introduced missingness to benchmark how well different methods could recover the original data's structure.

Key Insights and Ideas

  • FCS is Identifiable: The authors prove that if you impute one variable at a time, you can identify the correct distribution under MAR. This is a "win" for the MICE approach over complex joint models that often fail when distributions shift across missingness patterns.
  • The "Mean" is the Enemy: If you impute the mean (as missForest effectively does), you might get a great RMSE, but you'll fail at estimating quantiles or correlations. For general-purpose data cleaning, you want the imputed data to "look" like the real data, which requires drawing from a distribution.
  • Energy Distance > RMSE: The paper advocates for using Energy Distance to evaluate imputation. This metric measures how much the distribution of the imputed dataset differs from the ground truth. A method that gets the distribution right is far more useful for a downstream scientist than one that just minimizes squared error.

Key Results

  • mice-DRF Performance: In complex scenarios with nonlinearities, mice-DRF and mice-cart (using classification and regression trees) significantly outperformed deep learning methods like GAIN and MIWAE in terms of energy distance.
  • The missForest Paradox: missForest often achieved the best (lowest) RMSE but performed poorly on distributional tasks. It "shrank" the data toward the mean, making it a poor choice for researchers interested in the tails of a distribution or complex interactions.
  • FCS Robustness: The experiments showed that FCS methods are remarkably capable of reproducing the marginal distribution of variables even when the missingness mechanism is quite "difficult" (i.e., not EMAR).

Limitations and Future Work

  • Extrapolation: The authors admit that no nonparametric method (including mice-DRF) can handle Requirement 3—extreme covariate shifts—perfectly. If the "observed" data is in one range and the "missing" data is in a completely different range, these models struggle to extrapolate without parametric assumptions (like assuming a linear trend).
  • Computational Cost: Training a Distributional Random Forest for every variable in every iteration of a MICE loop is computationally expensive, especially for high-dimensional data.
  • Uncertainty Quantification: While the methods provide "good" imputations, the authors note that incorporating "model uncertainty" (accounting for the fact that the imputation model itself is an estimate) into these nonparametric methods remains an open challenge for future research.

Summary updated

Abstract

Missing values pose a persistent challenge in modern data science. Consequently, there is an ever-growing number of publications introducing new imputation methods in various fields. The present paper attempts to take a step back and provide a more systematic analysis. Starting from an in-depth discussion of the Missing at Random (MAR) condition for nonparametric imputation, we first investigate whether the widely used fully conditional specification (FCS) approach indeed identifies the correct conditional distributions. Based on this analysis, we propose three essential properties an ideal imputation method should meet, thus enabling a more principled evaluation of existing methods and more targeted development of new methods. In particular, we introduce a new imputation method, denoted mice-DRF, that meets two out of the three criteria. We also discuss ways to compare imputation methods, based on distributional distances. Finally, numerical experiments illustrate the points made in this discussion.
BibTeX
@misc{naf2024what,
  author = {Jeffrey Näf and Erwan Scornet and Julie Josse},
  title = {{What Is a Good Imputation Under MAR Missingness?}},
  year = {2024},
  eprint = {2403.19196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.19196},
}
Figure