A kernel two-sample test with selection bias

Alexis Bellot, Mihaela van der SchaarUAI 2021

Abstract

Standard two-sample tests often fail when data is collected through a biased mechanism, leading researchers to see "differences" that are actually just artifacts of how the data was sampled. This paper introduces Weighted Maximum Mean Discrepancy (WMMD), a non-parametric hypothesis test that uses importance sampling weights within a kernel framework to "rebalance" groups, allowing for a fair comparison even in the presence of selection bias.

The Challenge of Selection Bias

In a perfect world, comparing two populations—such as men and women in a medical trial—would involve drawing random, representative samples. In reality, researchers often face selection bias: individuals volunteer for studies, or certain demographics are more likely to be captured in a database.

Mathematically, this is represented as conditioning on a selection variable ($S=1$). If the factors influencing selection ($X$) are correlated with the outcome ($Y$), a standard two-sample test will likely reject the null hypothesis. This rejection is often not due to fundamental population differences, but because the samples are mismatched. For example, if highly educated men are more likely to participate in a study than highly educated women, a test might show a "gender difference" in income that is actually just a difference in education levels.

Causal graph illustrating selection bias where selection (S) is influenced by confounders (X) and group (T).

Methods: Weighted Maximum Mean Discrepancy (WMMD)

The authors build upon the Maximum Mean Discrepancy (MMD), a kernel-based metric that measures the distance between probability distributions by embedding them into a Reproducing Kernel Hilbert Space (RKHS).

The core innovation is the WMMD (Weighted MMD), which incorporates importance weights $w(x)$ to "re-weight" samples from one group so that the marginal distribution of their confounding variables matches the other group. Instead of estimating selection probabilities directly, which can be unstable, the authors use kernel mean matching. This involves finding weights that minimize the distance between the mean embeddings of the confounders in the two groups—a convex optimization problem that can be solved efficiently. The authors also derive the asymptotic theory for the null distribution, proving that the test remains consistent and maintains controlled Type I error rates even with data-dependent weights.

Experimental Validation

The paper evaluates WMMD using both synthetic high-dimensional data and the LaLonde Employment Dataset, a benchmark in causal inference.

In synthetic experiments where bias was manually injected, the standard MMD test exhibited a Type I error rate of nearly 1.0, failing to account for the bias. In contrast, WMMD successfully maintained the error rate near the target $\alpha = 0.05$. On the LaLonde dataset, while a standard MMD test suggested a massive difference between groups due to pre-existing demographic imbalances, WMMD correctly adjusted for these factors, providing a more honest assessment of the employment program's impact.

Overview of the adjustment process: from biased observational data to aligned income distributions using WMMD.

Key Insights and Limitations

WMMD treats selection bias as a "nuisance" that invalidates the coincidence axiom of standard MMD, and restores it through weighting. It is shown to be nearly as efficient as standard MMD when no bias exists, making it a viable "safety first" approach for researchers.

However, the test relies on the "ignorability" requirement: it assumes that all variables causing the bias have been measured. If the bias is driven by unobserved factors, the Type I error rate will increase, as the weighting cannot fully correct for hidden confounders. Future work may extend this weighting strategy to other structured spaces, such as graphs or time-series data.

Summary updated

BibTeX
@inproceedings{bellot2021kernel,
  author = {Alexis Bellot and Mihaela van der Schaar},
  title = {{A kernel two-sample test with selection bias}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
}
Figure