Sharing Pattern Submodels for Prediction with Missing Values

Lena Stempfle, Ashkan Panahi, Fredrik D. Johansson

Introduction

Missing data is a ubiquitous challenge in real-world machine learning, particularly within healthcare. Often, data is not missing at random but follows recurring patterns—for instance, due to varying clinical protocols across different hospitals. Standard approaches typically rely on imputation followed by regression or training independent models for each missingness pattern (Pattern Submodels, or PSM). However, imputation often fails when missingness is "informative" (Missing Not At Random), and PSM suffers from high variance when specific patterns have limited samples.

The SPSM Framework

The Sharing Pattern Submodel (SPSM) framework addresses this bias-variance tradeoff by allowing different missingness patterns to share information through a global "main model" while specializing only where necessary.

The SPSM framework decomposes coefficients into a shared main model (θ) and pattern-specific specializations (Δm).

In SPSM, the coefficients for any given missingness pattern $m$ are decomposed into two components:

  1. Main Model (θ): A global set of weights shared across all patterns, acting as an "anchor" for general trends.
  2. Pattern Specialization (Δm): Local adjustments unique to a specific pattern.

The prediction is calculated as $f_m(x) = (\theta_{\neg m} + \Delta_{\neg m})^\top x_{\neg m}$. To ensure parsimony and prevent over-specialization, an $\ell_1$ penalty is applied to the specialization terms ($\Delta_m$). This forces the model to use non-zero specialization coefficients only when they significantly improve predictive performance.

Experimental Evaluation

The authors evaluated SPSM against several baselines, including XGBoost, Multi-layer Perceptrons (MLP), Ridge regression, and standard PSM. Testing was conducted across synthetic data (simulating informative missingness) and real-world datasets:

  • ADNI: Neuroimaging and genetic data for Alzheimer’s prediction.
  • SUPPORT: Survival scores for hospitalized patients.
  • HOUSING: A non-healthcare dataset to test linear assumptions.

Key Results

A primary advantage of SPSM is its sample efficiency. By sharing information through the main model, SPSM stabilizes predictions for rare missingness patterns that lack sufficient data for independent modeling.

Performance comparison (R2) across varying dataset fractions. SPSM demonstrates superior sample efficiency, particularly when training data is scarce.

On the ADNI and SUPPORT datasets, SPSM matched or exceeded the performance of complex black-box models like MLPs and XGBoost. For example, in the ADNI regression task, SPSM achieved an $R^2$ of 0.66, equal to Ridge regression but with significantly higher interpretability. The sparse specialization terms allow researchers to identify exactly which features change in importance under specific missingness conditions.

Conclusion

SPSM bridges the gap between rigid global models and fragmented local ones. While currently limited to linear and logistic architectures, it provides a theoretically grounded and highly interpretable method for handling informative missingness. Future work may extend this regularized parameter-sharing philosophy to non-linear models.

Summary updated

Abstract

Missing values are unavoidable in many applications of machine learning and present challenges both during training and at test time. When variables are missing in recurring patterns, fitting separate pattern submodels have been proposed as a solution. However, fitting models independently does not make efficient use of all available data. Conversely, fitting a single shared model to the full data set relies on imputation which often leads to biased results when missingness depends on unobserved factors. We propose an alternative approach, called sharing pattern submodels (SPSM), which i) makes predictions that are robust to missing values at test time, ii) maintains or improves the predictive power of pattern submodels, and iii) has a short description, enabling improved interpretability. Parameter sharing is enforced through sparsity-inducing regularization which we prove leads to consistent estimation. Finally, we give conditions for when a sharing model is optimal, even when both missingness and the target outcome depend on unobserved variables. Classification and regression experiments on synthetic and real-world data sets demonstrate that our models achieve a favorable tradeoff between pattern specialization and information sharing.
BibTeX
@inproceedings{stempfle2023sharing,
  author = {Lena Stempfle and Ashkan Panahi and Fredrik D. Johansson},
  title = {{Sharing Pattern Submodels for Prediction with Missing Values}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i8.26179},
}
Figure