Benchmarking missing-values approaches for predictive models on health databases

Alexandre Perez-Lebel, Gaël Varoquaux, Marine Le Morvan, Julie Josse, Jean‐Baptiste Poline

TL;DR

This paper benchmarks various methods for handling missing values in predictive models on large health datasets. The key takeaway is that tree-based models with built-in missing value handling (MIA) outperform traditional imputation methods in both accuracy and computational efficiency, especially on large datasets.

What Problem Does This Paper Address?

In the era of big data, especially in healthcare, datasets are increasingly plagued by missing values. These missing values arise from various sources, such as incomplete patient records, differing data collection practices, or patient conditions. While traditional statistics has well-established methods for handling missing data for inferential purposes, their application in supervised machine learning for prediction is less understood. Existing benchmarks often focus on imputation quality rather than predictive performance, leaving practitioners without clear guidance on the best strategies for building accurate predictive models from incomplete health data. This paper aims to fill this gap by systematically evaluating different missing value handling techniques in the context of predictive modeling on real-world health datasets.

Methods

The authors compare two main categories of approaches:

  1. Imputation followed by prediction: This involves filling in missing values using various imputation strategies (mean, median, KNN, iterative imputation) and then training a predictive model on the completed dataset. A crucial variation explored is adding a "mask" or indicator column to signal which values were originally missing.
  2. Directly handling missing values: This approach leverages machine learning models that can natively process missing data. The primary method investigated is "Missing Incorporated Attribute" (MIA) within gradient-boosted trees.

The predictive model used across all methods is gradient-boosted trees, chosen for their state-of-the-art performance on tabular data. The study also includes a supplementary experiment comparing these methods with linear models.

Datasets

The benchmark was conducted on four diverse, real-world health databases:

  • Traumabase: Data from 20 French trauma centers on over 20,000 trauma cases.
  • UK Biobank (UKBB): A large prospective epidemiology cohort with health information on over 500,000 participants.
  • MIMIC-III: An intensive care unit (ICU) dataset containing de-identified health data from approximately 60,000 ICU admissions.
  • National Health Interview Survey (NHIS): A US national survey monitoring population health.

These databases were used to define 13 prediction tasks (10 classification, 3 regression) across various health outcomes. The study also explored the impact of dataset size by subsampling to 2,500, 10,000, 25,000, and 100,000 samples.

Key Insights and Ideas

  • Missingness is Informative: The addition of a "mask" (an indicator column for imputed values) consistently improved prediction accuracy. This suggests that the fact that a value is missing often carries predictive information, implying a "missing not at random" (MNAR) scenario, which deviates from assumptions often made in traditional imputation.
  • Native Handling Trumps Imputation (for Prediction): For predictive tasks, directly handling missing values with models like MIA in gradient-boosted trees is more efficient and often more accurate than complex imputation strategies.
  • Computational Cost Matters: Sophisticated imputation methods like MICE and KNN become computationally intractable on large datasets, making them impractical for many real-world applications.
  • Bagging Boosts Performance, But At a Price: Ensemble methods like bagging can improve prediction scores but significantly increase computational time, often by a factor of 100.
  • Good Imputation ≠ Good Prediction: The study challenges the intuition that better imputation automatically leads to better prediction. The predictive model's ability to leverage the imputed data and the nature of the missingness mechanism play crucial roles.

Key Results

  • MIA is the Champion: MIA, when used with gradient-boosted trees, consistently achieved top or near-top prediction performance across various dataset sizes and tasks, often outperforming state-of-the-art imputation methods.
  • Speed Advantage: MIA was significantly faster than most imputation methods, especially on larger datasets, offering a substantial computational cost saving. For example, MIA was up to 200 times faster than some bagged methods.
  • Masking is Crucial for Imputation: When imputation was used, adding a mask column to indicate imputed values was essential for good prediction performance.
  • Gradient-Boosted Trees Excel: Gradient-boosted trees generally outperformed linear models for predictive tasks, likely due to their ability to capture non-linear relationships in the data.
  • Bagging's Trade-off: While bagged methods like Iterative+mask+Bagging and MIA+Bagging achieved the best overall prediction ranks, their computational cost was prohibitive for large datasets.

Limitations and Future Work

  • Significance of Differences: Not all observed performance differences between methods were statistically significant, partly due to the limited number of datasets and the high computational cost of the benchmark.
  • Imputation Quality Not Assessed: The study focused solely on prediction performance and did not evaluate the accuracy of the imputed values themselves.
  • Hyperparameter Tuning: For computational reasons, hyperparameter tuning was not performed for the imputers themselves, which could potentially influence their performance.
  • Future Work: The authors suggest further work could explore joint model selection for imputation and supervised learning, and more extensive benchmarks with additional datasets and models. They also note that for very small datasets, simpler models like linear models might be sufficient.

Summary updated

Abstract

Native support for missing values in supervised machine learning predicts better than state-of-the-art imputation with much less computational cost. When using imputation, it is important to add indicator columns expressing which values have been imputed.
BibTeX
@inproceedings{perezlebel2022benchmarking,
  author = {Alexandre Perez-Lebel and Gaël Varoquaux and Marine Le Morvan and Julie Josse and Jean‐Baptiste Poline},
  title = {{Benchmarking missing-values approaches for predictive models on health databases}},
  booktitle = {GigaScience},
  year = {2022},
  doi = {10.1093/gigascience/giac013},
}
Figure