Mixed Matrix Completion in Complex Survey Sampling under Heterogeneous Missingness

Shu Yang, Xiaojun Mao, Hengfang Wang, Zhonglei Wang, Shu Yang 0006J. Comput. Graph. Stat 2024 · DOI

TL;DR

This paper tackles the challenging problem of completing missing data in large-scale surveys that use complex sampling designs and have mixed types of responses with non-uniform missingness. They propose a novel two-stage approach that models the missingness mechanism and then uses a low-rank matrix completion technique adapted for survey data, achieving robust and scalable results.

What Problem Does This Paper Address?

Modern surveys, like the National Health and Nutrition Examination Survey (NHANES), are increasingly collecting vast amounts of data using complex sampling designs and mixed-type questionnaires. A significant challenge in analyzing this data is the ubiquitous presence of missing values, which often occur in non-random patterns (heterogeneous missingness). Ignoring these missing values or using standard imputation methods can lead to biased estimates. Traditional matrix completion methods, which leverage the low-rank structure of data, often assume uniform missingness and don't account for the complexities of survey sampling, leading to potential inaccuracies. This paper aims to bridge this gap by developing a method that can effectively handle both complex survey sampling and heterogeneous missingness for mixed-type data.

Methods

The authors propose a two-stage procedure:

  1. Modeling the Missing Mechanism: In the first stage, they use logistic regression to model the probability of an entry being missing, incorporating auxiliary variables (covariates) from the survey. This step is crucial for addressing the heterogeneous missingness.
  2. Low-Rank Matrix Completion with Survey Weights: In the second stage, they complete the missing entries by maximizing a weighted log-likelihood function. This function is adapted to account for the complex survey design by incorporating inverse probability weights. A low-rank constraint is imposed on the target parameter matrix, which is a common assumption in matrix completion. They also leverage the fact that different types of survey questions (e.g., binary, continuous) can be modeled using different exponential family distributions.

They develop a fast and scalable estimation algorithm, MMCSHM (Mixed Matrix Completion in Survey with Heterogeneous Missing), which uses an iterative shrinkage-thresholding approach (FISTA) and is shown to have sublinear convergence.

Datasets

  • Synthetic Data: Used for experimental validation of the proposed method.
  • 2015-2016 National Health and Nutrition Examination Survey (NHANES) data: This real-world dataset is used to apply and evaluate the proposed method. It comprises 5375 sampled units, 14 strata, 16 demographic covariates, and responses to 130 survey questions (57 binary, 73 continuous).

Key Insights and Ideas

  • Heterogeneous Missingness is Key: The paper emphasizes that assuming uniform missingness in survey data is often unrealistic and can lead to biased results. Their method explicitly models this heterogeneity using covariates.
  • Survey Design Matters: Ignoring sampling weights and complex survey designs in matrix completion can also introduce bias. The proposed method incorporates these design features through inverse probability weighting.
  • Flexibility for Mixed-Type Data: By using an exponential family framework, the method can handle diverse response types (binary, continuous, etc.) within the same matrix, a significant advantage over methods that assume a single distribution for the entire matrix.
  • Low-Rankness as a Unifying Principle: The assumption of low-rankness in the underlying data structure is leveraged to effectively impute missing values across different question types and sampling strata.

Key Results

  • Theoretical Guarantees: The paper provides rigorous theoretical guarantees for the proposed method, including an upper bound on the estimation error.
  • Algorithmic Efficiency: The MMCSHM algorithm is shown to achieve sublinear convergence, making it computationally feasible for large datasets.
  • Superior Performance in Simulations: Experimental results on synthetic data demonstrate that the proposed method outperforms other popular matrix completion techniques (Collective, Soft-Impute, Hot-Deck) in terms of relative error, especially under low response probabilities.
  • Robustness in Real Data: When applied to the NHANES dataset, the proposed method shows consistent results with other methods, particularly when response rates are high, and diverges less as response rates decrease, indicating its robustness.

Limitations and Future Work

  • Computational Complexity: While the algorithm is designed to be scalable, the computational complexity for a single iteration is still dependent on the dimensions of the data (O(n(D+L) min{n, D+L})).
  • Extension to Longitudinal Data: The authors suggest extending the methodology to handle longitudinal observations, which would involve tensor completion.
  • Non-Convex Factorization: For improved numerical performance, they propose exploring non-convex factorization methods for the parameter matrix.
  • Assumptions: The theoretical guarantees rely on certain assumptions about the data and missingness mechanism, which may not always hold in practice.

Summary updated

BibTeX
@article{yang2024mixed,
  author = {Shu Yang and Xiaojun Mao and Hengfang Wang and Zhonglei Wang and Shu Yang},
  title = {{Mixed Matrix Completion in Complex Survey Sampling under Heterogeneous Missingness}},
  journal = {J. Comput. Graph. Stat},
  year = {2024},
  doi = {10.1080/10618600.2024.2319154},
}
Figure