Semiparametric Inference for Nonmonotone Missing-Not-at-Random Data: The No Self-Censoring Model

Daniel Malinsky, Ilya Shpitser, Eric J. Tchetgen TchetgenJournal of the American Statistical Association 2020 · DOI

Abstract

We study the identification and estimation of statistical functionals of multivariate data missing non-monotonically and not-at-random, taking a semiparametric approach. Specifically, we assume that the missingness mechanism satisfies what has been previously called "no self-censoring" or "itemwise conditionally independent nonresponse," which roughly corresponds to the assumption that no partially-observed variable directly determines its own missingness status. We show that this assumption, combined with an odds ratio parameterization of the joint density, enables identification of functionals of interest, and we establish the semiparametric efficiency bound for the nonparametric model satisfying this assumption. We propose a practical augmented inverse probability weighted estimator, and in the setting with a (possibly high-dimensional) always-observed subset of covariates, our proposed estimator enjoys a certain double-robustness property. We explore the performance of our estimator with simulation experiments and on a previously-studied data set of HIV-positive mothers in Botswana.

Introduction

Missing data is a pervasive challenge in empirical research. While the "Missing At Random" (MAR) assumption—where missingness depends only on observed data—is common, it often fails in practice. In many sensitive contexts, such as clinical trials or economic surveys, data is "Missing Not At Random" (MNAR), meaning the probability of a value being missing depends on the value itself. Handling MNAR data is notoriously difficult, especially when the missingness pattern is non-monotone (e.g., intermittent missingness in longitudinal studies).

This paper introduces a semiparametric framework for these complex scenarios by leveraging the "no self-censoring" assumption. This approach provides a robust alternative to traditional methods that often require restrictive parametric assumptions or fail to handle non-monotone patterns effectively.

The No Self-Censoring Model

The core of this framework is the "no self-censoring" (or itemwise conditionally independent nonresponse) assumption. Formally, this assumes that for any variable $L_j$, its missingness indicator $R_j$ is conditionally independent of the value $L_j$ itself, given all other variables (both observed and missing) and their indicators. In simpler terms, while variable $A$ might influence whether variable $B$ is missing, $A$ does not directly cause its own missingness.

Graphical representation of the no self-censoring model for three variables. Note the absence of direct edges from $L_j$ to $R_j$, illustrating the core assumption.

As shown in the figure above, the model can be represented as a chain graph or a directed acyclic graph (DAG) where no direct arrow exists between a variable $L_j$ and its corresponding indicator $R_j$. This graphical clarity allows researchers to assess the plausibility of the assumption based on domain knowledge.

Semiparametric Estimation and Double Robustness

To identify and estimate parameters under this model, the authors employ an odds ratio (OR) parameterization of the joint density. This technique decomposes the complex joint distribution into variationally independent components, bypassing the "curse of dimensionality" typically associated with MNAR models.

The authors derive the Efficient Influence Function (EIF) for this model, which represents the optimal estimator in terms of variance. Based on this, they propose an Augmented Inverse Probability Weighted (AIPW) estimator. A key advantage of this estimator is its double robustness: if a subset of covariates is always observed, the estimator remains consistent if either the model for the missingness mechanism or the model for the data distribution is correctly specified.

Empirical Validation

The proposed method was tested through both simulations and a real-world application:

  1. Simulations: The AIPW estimator outperformed standard techniques like Multiple Imputation by Chained Equations (MICE) in MNAR scenarios, showing significantly lower bias and higher precision.
  2. HIV Research in Botswana: The model was applied to a dataset of 9,711 HIV-positive mothers. In this high-stakes setting, HAART exposure and CD4+ counts were frequently missing. The AIPW estimator successfully identified a significant association between HAART continuation and preterm delivery for mothers with low CD4+ counts—an association that other MNAR methods failed to detect.

Conclusion

The "no self-censoring" model offers a powerful middle ground for missing data analysis. It is flexible enough to accommodate non-monotone MNAR data while providing the mathematical structure needed for robust, semiparametric inference. By combining graphical logic with doubly-robust estimation, this framework provides a vital tool for researchers working with complex, incomplete datasets in public health and beyond.

Summary updated

Abstract

We study the identification and estimation of statistical functionals of multivariate data missing non-monotonically and not-at-random, taking a semiparametric approach. Specifically, we assume that the missingness mechanism satisfies what has been previously called "no self-censoring" or "itemwise conditionally independent nonresponse," which roughly corresponds to the assumption that no partially-observed variable directly determines its own missingness status. We show that this assumption, combined with an odds ratio parameterization of the joint density, enables identification of functionals of interest, and we establish the semiparametric efficiency bound for the nonparametric model satisfying this assumption. We propose a practical augmented inverse probability weighted estimator, and in the setting with a (possibly high-dimensional) always-observed subset of covariates, our proposed estimator enjoys a certain double-robustness property. We explore the performance of our estimator with simulation experiments and on a previously-studied data set of HIV-positive mothers in Botswana.
BibTeX
@article{malinsky2020semiparametric,
  author = {Daniel Malinsky and Ilya Shpitser and Eric J. Tchetgen Tchetgen},
  title = {{Semiparametric Inference for Nonmonotone Missing-Not-at-Random Data: The No Self-Censoring Model}},
  journal = {Journal of the American Statistical Association},
  year = {2020},
  doi = {10.1080/01621459.2020.1862669},
}
Figure