Online Missing Value Imputation and Correlation Change Detection for Mixed-type Data via Gaussian Copula

Yuxuan Zhao, Eric Landgrebe, Eliot Shekhtman, Madeleine Udell

Abstract

Real-world streaming data is characterized by missing values, mixed data types, and shifting distributions over time. We present an online Expectation-Maximization (EM) algorithm based on the Gaussian copula to address these challenges. Our approach enables real-time missing value imputation and change point detection without the need for the rank-based hyperparameter tuning common in matrix factorization methods. Experiments on synthetic and real-world datasets demonstrate that our method is faster, more robust to concept drift, and more stable than existing state-of-the-art techniques.

Introduction

Many machine learning workflows require complete datasets, making missing value imputation a critical preprocessing step. While offline imputation is well-studied, performing it in an online streaming context—where data arrives from sensors or ongoing surveys—is significantly more difficult.

Most existing online imputation methods rely on Low Rank Matrix Completion (LRMC). However, LRMC struggles with "long skinny" datasets (matrices with many rows but few columns) because the low-rank assumption often breaks down. Furthermore, real-world data is frequently "mixed," containing a blend of continuous, ordinal, and binary variables. Finally, streaming data is subject to "concept drift," where underlying relationships between variables change. Detecting these shifts (Change Point Detection, or CPD) is notoriously difficult when the incoming data stream is riddled with missing values.

Methodology

The proposed method builds on the Gaussian Copula model. A copula separates the marginal distributions of individual variables from their dependency structure. It assumes that each data point is generated by drawing from a latent Gaussian vector, which is then transformed to match the observed variables (e.g., thresholding a continuous latent variable to create an ordinal rating).

Gaussian copula mapping between observed mixed-type data and latent Gaussian space.

To make this model viable for streaming data, we introduce three key components:

  1. Online Marginal Estimation: A sliding window approach tracks recent observations to estimate the marginal distribution of each column.
  2. Online EM for Correlation: Instead of recalculating the correlation matrix over all historical data, we use an online EM algorithm. It updates the correlation matrix by taking a weighted average of the previous estimate and the estimate from the newest batch of data.
  3. Online Change Point Detection: We propose a Monte Carlo hypothesis test that measures the distance (using the Frobenius norm) between the old correlation matrix and the newly updated one. The algorithm flags a change point if the update magnitude exceeds a simulated threshold.

Experiments and Key Results

We evaluated the algorithm using synthetic mixed-type data, the MovieLens 1M dataset, and Dow Jones Industrial Average (DJIA) stock data.

Imputation error (SMAE) across mixed data types and change point tracking on synthetic data.

Performance on Mixed Data

By mapping all variables to a latent Gaussian space, the copula naturally handles mixed data types. As shown in Figure 3, the Online EM method maintains low Scaled Mean Absolute Error (SMAE) across continuous, ordinal, and binary columns, even when faced with injected change points. Matrix factorization methods like GROUSE often force continuity assumptions that lead to poor performance on discrete data.

Robustness and Stability

A major advantage of the Gaussian copula method is its immunity to the "rank trap." Methods like GROUSE or online KFMC require a pre-specified rank; if this guess is wrong or the true rank changes, the model can diverge. In real-world tests on DJIA stock prices, our Online EM maintained stable imputation errors over 5,000 trading days, while competing methods eventually failed completely.

Speed and Change Point Detection

On the MovieLens dataset, a mini-batch version of the EM algorithm achieved the same accuracy as state-of-the-art offline EM but was approximately 7x faster. Additionally, the integrated CPD mechanism successfully detected distribution shifts in synthetic data, providing a highly accurate signal derived directly from the EM update process.

Conclusion

Our online Gaussian copula EM algorithm provides a robust, fast, and parameter-light framework for real-time data imputation and monitoring. Future work will focus on improving the computational efficiency of the Monte Carlo CPD test and developing methods to explicitly detect changes in marginal distributions alongside correlation shifts.

Summary updated

BibTeX
@inproceedings{zhao2022online,
  author = {Yuxuan Zhao and Eric Landgrebe and Eliot Shekhtman and Madeleine Udell},
  title = {{Online Missing Value Imputation and Correlation Change Detection for Mixed-type Data via Gaussian Copula}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20906},
}
Figure