Do we Need Dozens of Methods for Real World Missing Value Imputation?

Krystyna Grzesiak, Christophe Muller, Julie Josse, Jeffrey NäfarXiv 2025 · DOI · arXiv

TL;DR

Despite the explosion of complex deep learning models for data imputation, classic iterative methods still reign supreme. This large-scale benchmark of 73 algorithms across 24 datasets reveals that "simple" chained equations—specifically those in the mice R package—better preserve the underlying data distribution than fancy neural networks like GANs or VAEs.

What Problem Does This Paper Address?

Missing data is a pervasive headache in data science, but the way we evaluate "solutions" is often fundamentally flawed. Most researchers use pointwise metrics like Root Mean Squared Error (RMSE) to judge how well an algorithm fills in the blanks.

The authors argue that imputation is not prediction. If you have a missing value and you fill it with the conditional mean (the most "accurate" point prediction), you effectively shrink the variance of your dataset and destroy the relationships between variables. This leads to biased downstream analyses. The paper seeks to answer: which methods actually recover the distribution of the data, and do they work on real-world missingness where we don't have a "ground truth" to compare against?

Methods

The researchers conducted a massive systematic benchmark of 73 imputation algorithms. They categorized these into two main camps:

  1. Joint Modeling: Imputing data using a single model (e.g., VAEs, GANs, or parametric distributions).
  2. Fully Conditional Specification (FCS): Also known as "chained equations," where a separate model is trained for every single variable.

To evaluate them, they moved away from RMSE and used:

  • Energy Distance: A metric that measures the distance between two distributions. This was used for synthetic missingness where the "true" values were known.
  • I-Scores (Imputation Scores): A novel "proper scoring rule" that allows researchers to rank imputation quality on real-world datasets where the true values are unknown, provided a "Missing at Random" (MAR) assumption holds.

Datasets

The study utilized 24 different datasets ranging from 1,080 to 2.9 million elements. These included:

  • Purely Numerical Data: Standard benchmarks like airfoil, concrete, and yeast.
  • Mixed Data: Datasets containing both numerical and categorical variables (e.g., adult, diabetes, heart disease).
  • Real-World Missingness: 13 datasets with naturally occurring missing values (e.g., boys, colic, soybean), providing a much-needed reality check for algorithms usually tested only on artificial "amputated" data.

Key Insights and Ideas

  • The RMSE Paradox: The authors demonstrate that a method can have a "better" (lower) RMSE while visually and statistically ruining the dataset's geometry. For example, regression imputation scores better on RMSE than Gaussian imputation, but it creates a straight line of points that biases any subsequent correlation analysis.
  • Deep Learning Underperforms: Despite their popularity in recent literature, deep learning-based joint methods like GAIN, MIWAE, and miracle consistently underperformed. They were often less stable and less accurate at recovering distributions than iterative tree-based methods.
  • Stability is a Major Hurdle: Nearly half (30 out of 70) of the methods encountered computational errors or timed out (10-hour limit) on at least one scenario. The mice framework proved to be among the most robust.
  • The "Power of Four": The analysis suggests we don't need dozens of methods. A small set of four complementary methods—mice_cart, hyperimpute, aregImpute, and hyperimpute_em—provided over 90% coverage of the top-3 rankings across all scenarios.

Key Results

  • Numerical Winner: For purely numerical data, mice_cart (Classification and Regression Trees) was the clear champion, achieving a median rank of 4 across all tests.
  • Mixed Data Winner: In scenarios with categorical variables, mice_cart and mice_pmm (Predictive Mean Matching) remained the top performers.
  • Real-World King: When tested on real-world missingness using I-Scores, mice_rf (Random Forest) emerged as the superior choice. The authors suggest this is because the ensemble nature of Random Forests better captures the uncertainty and variability inherent in real-world data draws.
  • Iterative Superiority: Overall, FCS (iterative) methods overwhelmingly outperformed joint modeling approaches.

Limitations and Future Work

The authors acknowledge several constraints:

  • Dimensionality: The datasets were relatively low-dimensional (maximum 296 columns). The computational cost of FCS methods might "explode" on high-dimensional genomic or sensor data.
  • Data Structure: The study focused on i.i.d. (independent and identically distributed) data. It did not address time-series or panel data, where temporal dependencies change the imputation game entirely.
  • Implementation Speed: While mice is the most accurate, it is implemented in base R and can be slow. Python-based alternatives like hyperimpute are faster but showed some performance gaps on categorical data.

Future work could involve extending this distributional benchmarking to high-dimensional settings and exploring how to optimize the speed of these top-performing iterative algorithms.

Summary updated

Abstract

Missing values pose a persistent challenge in modern data science. Consequently, there is an ever-growing number of publications introducing new imputation methods in various fields. While many studies compare imputation approaches, they often focus on a limited subset of algorithms and evaluate performance primarily through pointwise metrics such as RMSE, which are not suitable to measure the preservation of the true data distribution. In this work, we provide a systematic benchmarking method based on the idea of treating imputation as a distributional prediction task. We consider a large number of algorithms and, for the first time, evaluate them not only on synthetic missing mechanisms, but also on real-world missingness scenarios, using the concept of Imputation Scores. Finally, while the focus of previous benchmark has often been on numerical data, we also consider mixed data sets in our study. The analysis overwhelmingly confirms the superiority of iterative imputation algorithms, especially the methods implemented in the mice R package.
BibTeX
@misc{grzesiak2025do,
  author = {Krystyna Grzesiak and Christophe Muller and Julie Josse and Jeffrey Näf},
  title = {{Do we Need Dozens of Methods for Real World Missing Value Imputation?}},
  year = {2025},
  eprint = {2511.04833},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.04833},
  howpublished = {arXiv},
}
Figure