Inference and Imputation with Informative Missingness

721 concepts452 papers3 proposals

When data goes missing, we often treat it as an unfortunate accident—a random glitch in our data collection pipeline. In classical statistics, this is known as data that is Missing Completely at Random (MCAR) or Missing at Random (MAR). Under these assumptions, the reason data is missing is independent of the missing values themselves. We can safely ignore the mechanism that caused the data to vanish, fill in the blanks using standard imputation, or simply drop incomplete rows.

But in the real world, missingness is rarely accidental; it is often highly strategic or systematic. Consider a survey on personal wealth where high-income individuals refuse to disclose their earnings, or an electronic health record where doctors only order specific blood tests if they suspect a patient is critically ill. This is Missing Not at Random (MNAR) data, also known as informative or non-ignorable missingness.

The fundamental challenge of MNAR data is an entanglement problem: the probability that a value is missing depends directly on the true, unobserved value itself. This creates a severe mathematical barrier. As noted by Jarrett et al., MNAR scenarios are theoretically unidentifiable without incorporating specific domain knowledge or structural assumptions. If you ignore this mechanism and apply standard tools, your models will suffer from severe selection bias. For example, Zhang et al. demonstrate that dropping incomplete rows ("complete case analysis") under non-random missingness severely distorts algorithmic fairness assessments, while Zhang et al. show that standard causal discovery algorithms will "hallucinate" false causal relationships by mistaking missingness patterns for actual physical links.

To solve this, a growing body of research focuses on explicitly modeling the missingness mechanism, transforming the "gaps" in our data from a nuisance into a structured source of information.


Causal Missingness Graphs

How do we know if a dataset plagued by informative missingness can even be salvaged? To answer this, researchers turn to causal graphical models.

Mohan et al. formalized this approach by introducing Missingness Graphs (m-graphs). By representing both the underlying variables and their missingness indicators as nodes in a Directed Acyclic Graph (DAG), we can visually and mathematically map out why data goes missing.

  True Value (X)  -------->  Missingness Indicator (M_x)
        |                               ^
        v                               |
  Observed Value (X*) ------------------+

In an m-graph, if an arrow points directly from the true variable $X$ to its own missingness indicator $M_x$, the data is MNAR. The power of this framework lies in its ability to determine recoverability—proving whether an unbiased joint distribution can be reconstructed from the observed data. Building on this graphical foundation, Tian et al. developed the M-Adjustment and MS-Adjustment criteria. These graph-based rules allow algorithms to scan a network structure, identify valid adjustment sets of variables, and systematically block the biasing paths caused by self-censoring and informative missingness.


Bypassing the Mechanism with Shadow Variables

Explicitly modeling a complex, non-random missingness mechanism can be incredibly difficult, especially when the true data distribution is unknown. An elegant alternative is to bypass modeling the mechanism entirely by leveraging shadow variables (or instrumental variables).

The intuition here is to find an auxiliary variable that is related to the true underlying value but is conditionally independent of the missingness process itself. Zhao et al. introduced a "nuisance-free" conditional likelihood procedure that uses shadow variables to achieve valid statistical inference without ever having to specify or estimate the exact missingness function.

This concept is particularly powerful in self-censoring scenarios. For instance, Chen demonstrated how a randomized incentive (like a gift card or a specific survey mode) can act as an anchor. Because the incentive alters whether a participant responds but is completely unrelated to their hidden true attribute, researchers can use odds-ratio factorization and inverse probability weighting to mathematically untangle the true data distribution from the participant's propensity to hide their data.


Treating the Missingness Mask as a Signal

Traditional machine learning pipelines treat the missingness mask—the matrix of 1s and 0s indicating whether data was observed or missing—as a secondary byproduct to be erased via imputation. However, when data is missing non-randomly, the absence of data is itself a first-class signal.

Instead of trying to fill in the blanks first, modern architectures process the data and the missingness mask in parallel. Chen et al. designed a deep generative model (GNR) that uses a "conjunction model" to treat the data and the mask as two equal modalities. By forcing the network to learn the joint distribution of the content and its visibility simultaneously, the model gains a structural "understanding" of why the data is missing, leading to highly accurate, unbiased imputations.

Treating the mask as a feature also prevents models from failing when deployed in new environments. As Swamy et al. and Feng et al. demonstrate, when a model explicitly captures the signal within the gaps, it stops relying on missingness as a shortcut predictive feature. This architectural shift ensures that if data availability patterns change downstream, the model's predictive accuracy and demographic fairness remain stable.


Exploiting Geometric and Low-Rank Structures

When domain knowledge or shadow variables are unavailable, we can still overcome non-random missingness by exploiting the mathematical and geometric structures inherent in the data matrix itself.

Standard matrix completion algorithms assume that entries are missing uniformly at random. To handle informative missingness, Needell introduced algorithms that regularize unobserved entries based on structured observations—such as assuming that unobserved entries in a recommendation system are systematically skewed toward low ratings. Similarly, Boyer developed low-rank matrix completion frameworks that combine explicit Expectation-Maximization (EM) algorithms with implicit mask-feature learning to correct for extreme-value selection biases.

Remarkably, local geometric properties can also bypass the need to model the missingness mechanism. Sadhukhan et al. proved that local neighborhood averaging methods, such as the Two-Sided Nearest Neighbors (TS-NN) algorithm, are inherently robust to MNAR and deterministic missingness patterns. As long as the underlying data manifold is sufficiently smooth and enough local neighbors can be identified, the estimation error remains tightly bounded—proving that local geometric structure can naturally neutralize the biases of non-random data loss.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_non_random_missingness_f39c1e73,
  title = {non-random missingness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-random-missingness/rdr_f39c1e735bae40a9a35399f06a35960b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_non_random_missingness_f39c1e73_v1,
  title = {non-random missingness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/non-random-missingness/rdr_f39c1e735bae40a9a35399f06a35960b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake