Abstract: We study the problem of estimating a functional or a parameter in the context where outcome is subject to nonignorable missingness. We completely avoid modeling the regression relation, while allowing the propensity to be modeled by a semiparametric logistic relation where the dependence on covariates is unspecified. We discover a surprising phenomenon in that the estimation of the parameter in the propensity model as well as the functional estimation can be carried out without assessing the missingness dependence on covariates. This allows us to propose a general class of estimators for both model parameter estimation and functional estimation, including estimating the outcome mean. The robustness of the estimators are nonstandard and are established rigorously through theoretical derivations, and are supported by simulations and a data application.
Introduction
In statistical analysis, missing data is a pervasive challenge. While many standard techniques assume that data is "Missing at Random" (MAR)—where the probability of missingness depends only on observed variables—real-world scenarios often involve "nonignorable missingness." In these cases, the probability of a value being missing (the propensity) is directly tied to the value itself. For example, individuals with higher incomes may be less likely to report their earnings in a survey.
Addressing nonignorable missingness typically requires modeling the propensity score. However, these models often involve a complex "nuisance" component: an unknown function that describes how background covariates influence the probability of missingness. Traditional semiparametric methods require estimating this function through computationally intensive techniques like kernel estimation or profiling. This paper introduces a method to bypass this requirement entirely, allowing for consistent estimation even when the covariate-missingness relationship is misspecified.
Methodology: The "Working Model" Trick
The research focuses on a semiparametric logistic propensity model. The core innovation lies in the application of geometric semiparametric theory to derive an efficient score for the parameters of interest.
The authors demonstrate a "surprising phenomenon": the estimation of the propensity model parameters and the final functional (such as the outcome mean) can be performed without accurately assessing how missingness depends on covariates. They prove a Main Theorem showing that if an arbitrary "working model" is used for the unknown covariate function, the resulting estimating equations remain valid.
This creates a form of nonstandard robustness. Unlike traditional "double robustness," which requires at least one of two models (the outcome model or the propensity model) to be correctly specified, this approach succeeds as long as the parametric structure of the propensity model holds, regardless of whether the nonparametric covariate component is correct. This allows researchers to use a simple, even incorrect, placeholder for the unknown function without biasing the final results.
Empirical Results
Computational Efficiency
The most striking advantage of this method is its computational speed. By avoiding the iterative estimation of the nonparametric nuisance function, the proposed method is orders of magnitude faster than existing state-of-the-art approaches.
In simulations involving two-dimensional covariates, the new method completed 1,000 replicates in less than one minute. In contrast, the method proposed by Shao & Wang (2016) required 25 minutes for a small sample size ($N=200$) and was projected to take approximately 16 hours for a larger sample ($N=1,000$). This makes the new approach feasible for large-scale datasets where previous semiparametric methods were computationally prohibitive.
Accuracy and Robustness
Simulations confirmed that even when the "working model" for the unknown covariate function was intentionally misspecified, the estimation bias remained near zero. While there is a slight loss in efficiency (increased variance) if the working model is far from the truth, the estimator remains consistent.
Real-World Application: KLIPS Data
The researchers applied their method to the Korean Labor & Income Panel Study (KLIPS) to estimate the mean monthly income of regular wage earners. In this dataset, approximately 35% of income data was missing. A "naive" analysis that ignored the nonignorable missingness severely overestimated the mean income. The proposed method successfully corrected this bias, providing results consistent with other complex semiparametric models but with significantly less computational overhead.
Discussion and Limitations
While the method offers a powerful shortcut, it relies on the existence of a "nonresponse instrument"—a variable that influences the outcome but does not directly affect the probability of missingness (or vice versa). Identifying such instruments is a common challenge in observational studies. Additionally, if the chosen working model for the nuisance function is extremely poor, the standard errors of the estimates will increase, representing a trade-off between computational ease and statistical efficiency.
In conclusion, this research provides a robust and highly efficient framework for handling nonignorable missingness. By proving that the unknown covariate relationship can be effectively ignored during the estimation process, the authors have removed a significant barrier to the practical application of semiparametric propensity models in data science.
Summary updated