TL;DR
Despite the explosion of complex deep learning models for data imputation, classic iterative methods still reign supreme. This large-scale benchmark of 73 algorithms across 24 datasets reveals that "simple" chained equations—specifically those in the mice R package—better preserve the underlying data distribution than fancy neural networks like GANs or VAEs.
What Problem Does This Paper Address?
Missing data is a pervasive headache in data science, but the way we evaluate "solutions" is often fundamentally flawed. Most researchers use pointwise metrics like Root Mean Squared Error (RMSE) to judge how well an algorithm fills in the blanks.
The authors argue that imputation is not prediction. If you have a missing value and you fill it with the conditional mean (the most "accurate" point prediction), you effectively shrink the variance of your dataset and destroy the relationships between variables. This leads to biased downstream analyses. The paper seeks to answer: which methods actually recover the distribution of the data, and do they work on real-world missingness where we don't have a "ground truth" to compare against?
Methods
The researchers conducted a massive systematic benchmark of 73 imputation algorithms. They categorized these into two main camps:
- Joint Modeling: Imputing data using a single model (e.g., VAEs, GANs, or parametric distributions).
- Fully Conditional Specification (FCS): Also known as "chained equations," where a separate model is trained for every single variable.
To evaluate them, they moved away from RMSE and used:
- Energy Distance: A metric that measures the distance between two distributions. This was used for synthetic missingness where the "true" values were known.
- I-Scores (Imputation Scores): A novel "proper scoring rule" that allows researchers to rank imputation quality on real-world datasets where the true values are unknown, provided a "Missing at Random" (MAR) assumption holds.
Datasets
The study utilized 24 different datasets ranging from 1,080 to 2.9 million elements. These included:
- Purely Numerical Data: Standard benchmarks like
airfoil,concrete, andyeast. - Mixed Data: Datasets containing both numerical and categorical variables (e.g.,
adult,diabetes,heart disease). - Real-World Missingness: 13 datasets with naturally occurring missing values (e.g.,
boys,colic,soybean), providing a much-needed reality check for algorithms usually tested only on artificial "amputated" data.
Key Insights and Ideas
- The RMSE Paradox: The authors demonstrate that a method can have a "better" (lower) RMSE while visually and statistically ruining the dataset's geometry. For example, regression imputation scores better on RMSE than Gaussian imputation, but it creates a straight line of points that biases any subsequent correlation analysis.
- Deep Learning Underperforms: Despite their popularity in recent literature, deep learning-based joint methods like GAIN, MIWAE, and miracle consistently underperformed. They were often less stable and less accurate at recovering distributions than iterative tree-based methods.
- Stability is a Major Hurdle: Nearly half (30 out of 70) of the methods encountered computational errors or timed out (10-hour limit) on at least one scenario. The
miceframework proved to be among the most robust. - The "Power of Four": The analysis suggests we don't need dozens of methods. A small set of four complementary methods—
mice_cart,hyperimpute,aregImpute, andhyperimpute_em—provided over 90% coverage of the top-3 rankings across all scenarios.
Key Results
- Numerical Winner: For purely numerical data,
mice_cart(Classification and Regression Trees) was the clear champion, achieving a median rank of 4 across all tests. - Mixed Data Winner: In scenarios with categorical variables,
mice_cartandmice_pmm(Predictive Mean Matching) remained the top performers. - Real-World King: When tested on real-world missingness using I-Scores,
mice_rf(Random Forest) emerged as the superior choice. The authors suggest this is because the ensemble nature of Random Forests better captures the uncertainty and variability inherent in real-world data draws. - Iterative Superiority: Overall, FCS (iterative) methods overwhelmingly outperformed joint modeling approaches.
Limitations and Future Work
The authors acknowledge several constraints:
- Dimensionality: The datasets were relatively low-dimensional (maximum 296 columns). The computational cost of FCS methods might "explode" on high-dimensional genomic or sensor data.
- Data Structure: The study focused on i.i.d. (independent and identically distributed) data. It did not address time-series or panel data, where temporal dependencies change the imputation game entirely.
- Implementation Speed: While
miceis the most accurate, it is implemented in base R and can be slow. Python-based alternatives likehyperimputeare faster but showed some performance gaps on categorical data.
Future work could involve extending this distributional benchmarking to high-dimensional settings and exploring how to optimize the speed of these top-performing iterative algorithms.
Summary updated