Benchmarking missing-values approaches for predictive models on health databases

Alexandre Perez-Lebel, Gaël Varoquaux, Marine Le Morvan, Julie Josse, Jean-Baptiste Poline

Abstract

As databases grow larger, missing values become an inevitable challenge for predictive modeling. While traditional statistics focuses on inferential accuracy, modern machine learning prioritizes predictive performance. This paper presents a systematic benchmark of missing-value strategies using 520,000 CPU hours across massive health databases. We compare native support for missing values with various imputation techniques, finding that native support (Missing Incorporated in Attribute, or MIA) provides a fast, robust, and high-performing alternative to computationally expensive imputation methods.

The Challenge of Missing Data in Health ML

Missing data is a pervasive issue in healthcare, often resulting from varied clinical practices, equipment differences, or patient stability. In predictive modeling, the goal is to forecast outcomes like mortality or diagnosis from incomplete records. Historically, researchers have relied on "imputation"—filling in the blanks—but this can be slow and may inadvertently mask clinical signals. For instance, the absence of a test result can itself be an informative indicator of a patient's status (Missing Not At Random, or MNAR).

Methodology

The study evaluated 12 strategies using Gradient-Boosted Trees (specifically scikit-learn’s HistGradientBoosting) across four diverse health databases: Traumabase, UK Biobank, MIMIC-III, and NHIS. These datasets covered 13 different prediction tasks with sample sizes ranging from 2,500 to 100,000.

The strategies were categorized into:

  1. Single Imputation: Using simple statistics (Mean, Median) or conditional models (Iterative Imputer, KNN).
  2. Missingness Indicators ("Mask"): Adding binary columns to indicate which values were originally missing.
  3. Native Support (MIA): Modifying the tree-building algorithm to handle missing values directly by choosing the optimal split direction for them.
  4. Multiple Imputation via Bagging: Training ensembles on multiple imputed datasets to account for uncertainty.

Performance and Accuracy

A key finding was that adding a "mask" improved nearly every imputation method, confirming that health data is frequently MNAR. However, sophisticated imputation did not always translate to superior prediction.

Relative prediction scores across different sample sizes and imputation strategies. MIA and Bagging methods consistently rank among the top performers.

As shown in the performance analysis, the best overall predictive scores were achieved by Iterative Imputation + Mask + Bagging. However, the MIA strategy (native support) performed remarkably well, consistently ranking near the top without the need for additional indicator columns or complex pre-processing.

Computational Efficiency

While bagging methods offered the highest accuracy, they came with a massive computational penalty. On large datasets ($n=100,000$), methods like KNN became intractable, while bagging methods were 100 to 200 times slower than their non-bagged counterparts.

Relative total training time for various missing-value strategies. Note the 100x-200x jump in time for bagging methods compared to native MIA support.

MIA emerged as the most practical winner, finishing in hours where other methods took days or weeks. It naturally captures the information provided by a "mask" without doubling the feature space, making it the most efficient "bang for your buck" in large-scale health informatics.

Conclusion

For predictive modeling on large-scale health data, native support for missing values (MIA) in tree-based models is the gold standard. It offers a robust, fast, and high-performing solution that avoids the pitfalls and computational costs of complex imputation. When imputation is necessary, researchers should always include missingness indicators to capture the informative nature of missing data.

Summary updated

Abstract

BACKGROUND: As databases grow larger, it becomes harder to fully control their collection, and they frequently come with missing values: incomplete observations. These large databases are well suited to train machine-learning models, for instance for forecasting or to extract biomarkers in biomedical settings. Such predictive approaches can use discriminative -- rather than generative -- modeling, and thus open the door to new missing-values strategies. Yet existing empirical evaluations of strategies to handle missing values have focused on inferential statistics. RESULTS: Here we conduct a systematic benchmark of missing-values strategies in predictive models with a focus on large health databases: four electronic health record datasets, a population brain imaging one, a health survey and two intensive care ones. Using gradient-boosted trees, we compare native support for missing values with simple and state-of-the-art imputation prior to learning. We investigate prediction accuracy and computational time. For prediction after imputation, we find that adding an indicator to express which values have been imputed is important, suggesting that the data are missing not at random. Elaborate missing values imputation can improve prediction compared to simple strategies but requires longer computational time on large data. Learning trees that model missing values-with missing incorporated attribute-leads to robust, fast, and well-performing predictive modeling. CONCLUSIONS: Native support for missing values in supervised machine learning predicts better than state-of-the-art imputation with much less computational cost. When using imputation, it is important to add indicator columns expressing which values have been imputed.
BibTeX
@inproceedings{perezlebel2022benchmarking,
  author = {Alexandre Perez-Lebel and Gaël Varoquaux and Marine Le Morvan and Julie Josse and Jean-Baptiste Poline},
  title = {{Benchmarking missing-values approaches for predictive models on health databases}},
  booktitle = {GigaScience},
  year = {2022},
  doi = {10.1093/gigascience/giac013},
}
Figure