Correcting sample selection bias by unlabeled data

Jiancong Huang, Aj Smola, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, Jiayuan Huang, Alex SmolaANU 2007 · OpenAlex

Abstract

We consider the scenario where training and test data are drawn from different distributions, commonly referred to as sample selection bias. Most algorithms for this setting try to first recover sampling distributions and then make appropriate corrections based on the distribution estimate. We present a nonparametric method which directly produces resampling weights without distribution estimation. Our method works by matching distributions between training and testing sets in feature space. Experimental results demonstrate that our method works well in practice.

Introduction

In standard machine learning, we typically operate under the i.i.d. assumption: that training and test data are drawn independently from the same underlying distribution. However, in real-world applications, this assumption is frequently violated. This discrepancy, known as sample selection bias or covariate shift, occurs when the training data is not representative of the population on which the model will be deployed. For instance, a medical diagnostic tool trained on data from a specific hospital may fail when applied to a broader, more diverse population.

The Challenge of Density Estimation

To correct for this bias, the traditional approach involves reweighting training examples using the ratio of the test density to the training density:

$$\beta(x) = \frac{P_{test}(x)}{P_{train}(x)}$$

While theoretically sound, this approach is practically difficult. Estimating probability densities in high-dimensional spaces is notoriously unstable. Small errors in the denominator ($P_{train}$) can lead to massive fluctuations in the weights, often resulting in poor model performance.

Methodology: Kernel Mean Matching (KMM)

This paper introduces Kernel Mean Matching (KMM), a non-parametric approach that bypasses density estimation entirely. Instead of calculating $P(x)$, KMM leverages the properties of Reproducing Kernel Hilbert Spaces (RKHS). The core idea is that if the means of two distributions match in a sufficiently rich (universal) feature space, the distributions themselves must be identical.

The KMM Workflow

  1. Feature Mapping: Data points are mapped into a high-dimensional feature space using a universal kernel, such as the Gaussian RBF kernel.
  2. Discrepancy Minimization: The algorithm finds weights ($\beta$) for the training points that minimize the distance between the weighted training mean and the unweighted test mean in the feature space.
  3. Optimization: This is formulated as a Quadratic Programming (QP) problem, ensuring a global optimum is found efficiently.
  4. Constraints: Weights are bounded ($\beta \in [0, B]$) to ensure the resulting distribution remains stable.
Figure 1: Comparison of various fitting models on a biased regression task. The KMM-weighted model (green dashed line) significantly outperforms standard OLS (black dashed line) by accurately tracking the true underlying model (blue line) in the biased region.

Experimental Validation

The authors evaluated KMM across diverse scenarios, including synthetic regression tasks, the UCI Breast Cancer dataset, and 18 other benchmark datasets. They also tested the method on real-world microarray data, where "batch effects" between different laboratories create natural distribution shifts.

Key Results and Insights

KMM demonstrated robust performance across nearly all test cases. In 13 out of 23 benchmark experiments, it was the top-performing method, consistently outperforming unweighted baselines.

Key Findings:

  • Direct Weighting Advantage: By avoiding the "middleman" of density estimation, KMM is less sensitive to the errors that typically plague importance sampling.
  • Robustness to Bias Types: While designed for covariate shift (where only the feature distribution $P(x)$ changes), KMM proved effective even when the label distribution $P(y|x)$ was biased.
  • Sample-Specific Optimization: Surprisingly, KMM occasionally outperformed reweighting using the actual known density ratio. This suggests that by optimizing for the specific finite samples available, KMM can be more precise than theoretical population-level corrections.
Figure 6: Classification performance on the breast cancer dataset. KMM consistently reduces test error across simple feature bias (a), joint feature bias (b), and label bias (c). Panel (d) shows that KMM-derived weights effectively track the true inverse sampling probabilities.

Conclusion and Future Work

Kernel Mean Matching offers a powerful, unsupervised way to handle sample selection bias using only unlabeled test data. While the method is computationally efficient for moderate datasets, future work may focus on scaling the quadratic programming step for extremely large-scale data. Additionally, refining heuristics for hyperparameter selection (such as the kernel width $\sigma$ and weight bound $B$) remains an area for further development. Despite these challenges, KMM represents a significant step forward in building machine learning models that are robust to real-world distribution shifts.

Summary updated

Abstract

We consider the scenario where training and test data are drawn from\ndifferent distributions, commonly referred to as sample\nselection bias. Most algorithms for this setting try to first\nrecover sampling distributions and then make appropriate corrections based on the\ndistribution estimate. We present a nonparametric method which\ndirectly produces resampling weights without distribution estimation.\nOur method works by matching distributions between training and\ntesting sets in feature space. Experimental results demonstrate that our method works well in\npractice.
BibTeX
@inproceedings{huang2007correcting,
  author = {Jiancong Huang and AJ Smola and Arthur Gretton and Karsten Borgwardt and Bernhard Schölkopf and Jiayuan Huang and Alex Smola},
  title = {{Correcting sample selection bias by unlabeled data}},
  booktitle = {ANU},
  year = {2007},
}
Figure