Deeply Learned Generalized Linear Models with Missing Data

David K. Lim, Naim U. Rashid, Junier B. Oliva, Joseph G. IbrahimJ. Comput. Graph. Stat 2022

TL;DR

This paper introduces dlglm, a novel deep learning architecture for generalized linear models that can handle missing data in both features and the response variable. It offers a flexible framework for modeling complex relationships and accounting for various missingness mechanisms, including the challenging "missing not at random" (MNAR) scenario, outperforming existing methods in simulations and real-world applications.

What Problem Does This Paper Address?

Many real-world datasets suffer from missing values, which pose a significant challenge for machine learning models, especially deep learning architectures. Traditional methods for handling missing data often struggle with complex relationships, scalability, or make restrictive assumptions about the missingness mechanism (e.g., assuming data is "missing at random"). Deep learning models, while powerful, are particularly sensitive to missing data, as it can disrupt the intricate layers of transformations. This paper addresses the critical need for a deep learning approach that can robustly handle missing data, particularly in the context of generalized linear models (GLMs), while accounting for potentially non-ignorable missingness patterns (MNAR).

Methods

The core contribution is the dlglm (Deeply-Learned GLM with Missing Data) architecture. It extends traditional GLMs by using a deep neural network to model the non-linear relationships between features and the response. Crucially, dlglm incorporates a "missingness network" that explicitly models the mechanism by which data becomes missing. This allows it to handle both ignorable (MCAR, MAR) and non-ignorable (MNAR) missingness.

Key methodological components include:

  • Deeply-Learned GLM: Replaces the linear predictor in a traditional GLM with the output of a neural network, allowing for complex, non-linear feature interactions.
  • Variational Inference and Autoencoders (VAE/IWAE): Used to learn approximate posterior distributions for missing variables and to model the covariate distribution p(X). This is essential for handling the uncertainty introduced by missing data.
  • Missingness Network: A dedicated neural network component trained to model the probability of missingness, which is critical for MNAR scenarios.
  • Selection Model Factorization: A statistical framework used to combine the model for the data and the model for the missingness mechanism.

The paper also introduces idlglm, a variant designed for ignorable missingness, which simplifies the model by not requiring an explicit missingness model.

Datasets

The paper evaluates dlglm and idlglm on:

  • Simulated Data: Generated synthetic datasets to systematically test performance under various missingness mechanisms (MCAR, MAR, MNAR) and data dimensions.
  • UCI Machine Learning Repository Datasets:
    • DRYBEAN
    • LETTER
    • SHUTTLE
    • CAREVALUATION
    • BANKNOTE
    • RED
    • WHITE
    • SPAM
  • Bank Marketing Dataset: A real-world dataset with inherent missingness, used for a practical application of the proposed methods.

No new benchmark datasets were introduced; existing, well-known datasets were utilized.

Key Insights and Ideas

  • MNAR is Hard, and DL Can Help: The paper demonstrates that explicitly modeling the missingness mechanism (MNAR) is crucial for accurate predictions and coefficient estimation, and that deep learning architectures are well-suited for this task.
  • Unified Framework: dlglm provides a single, end-to-end framework that handles missing data during training, eliminating the need for separate imputation steps before model fitting, which is a significant advantage over traditional "impute-then-regress" methods.
  • Flexibility is Key: The ability to model complex non-linear relationships and various missingness mechanisms makes dlglm adaptable to diverse real-world data challenges.
  • idlglm for Simplicity: When missingness can be assumed to be ignorable (MCAR/MAR), the idlglm variant offers a simpler, yet effective, alternative.

Key Results

  • Superior MNAR Performance: In simulations, dlglm consistently outperformed other methods (mean imputation, mice, miwae, notmiwae) in imputation, coefficient estimation, and prediction under MNAR missingness.
  • Competitive in MAR/MCAR: Under MAR and MCAR settings, dlglm performed comparably to or better than established methods like mice and idlglm.
  • Real-World Efficacy: On the Bank Marketing dataset, dlglm (and its variant dlglmy) showed strong prediction performance, particularly when accounting for the response variable in the missingness model.
  • Scalability: The deep learning approach allows dlglm to scale to large datasets and high-dimensional feature spaces.
  • Robustness: dlglm showed robustness to different missingness mechanisms and dataset characteristics.

Limitations and Future Work

  • Computational Complexity: The authors note that dlglm can be complex and may not perform optimally on very small sample sizes (suggesting a minimum of 10,000 samples).
  • Hyperparameter Tuning: Like most deep learning models, dlglm requires careful hyperparameter tuning, which can be computationally intensive.
  • Generalization to Multivariate Response: While the paper focuses on univariate response Y, the authors suggest that dlglm can be generalized to multivariate Y by extending the neural network architecture and accounting for correlations between responses. This is left as future work.
  • Identifiability: While the paper observed consistent convergence, a formal derivation of dlglm's identifiability is not the primary focus.

Summary updated

BibTeX
@article{lim2022deeply,
  author = {David K. Lim and Naim U. Rashid and Junier B. Oliva and Joseph G. Ibrahim},
  title = {{Deeply Learned Generalized Linear Models with Missing Data}},
  journal = {J. Comput. Graph. Stat.},
  year = {2022},
}
Figure