Balance-Subsampled Stable Prediction

Kun Kuang, Hengtao Zhang, Fei Wu, Yueting Zhuang, Aijun ZhangarXiv 2020 · arXiv

Abstract

Machine learning models often fail under distribution shift due to their tendency to exploit spurious correlations present in training data. We present Balance-Subsampled Stable Prediction (BSSP), a novel framework that utilizes Fractional Factorial Design (FFD) to subsample observational data into a balanced, deconfounded subset. By ensuring feature independence within the training subdata, BSSP forces models to focus on stable, causal relationships rather than selection-biased shortcuts. Experimental results on synthetic and real-world datasets, including advertising and biomedical telemonitoring, demonstrate that BSSP significantly improves prediction stability and accuracy across unknown test distributions.

Introduction

A fundamental assumption in standard machine learning is that training and test data are independent and identically distributed (i.e., the i.i.d. assumption). However, in real-world applications, sample selection bias often creates spurious correlations—relationships that exist in the training set but vanish or reverse in the test set. For instance, a model trained to recognize dogs primarily in grassy environments may fail when encountering a dog on a beach, having incorrectly identified "grass" as a predictive feature for "dog."

To address this, the BSSP framework treats observational data through the lens of experimental statistics, specifically borrowing from the theory of Fractional Factorial Design to eliminate confounding factors at the source.

The Challenge of Spurious Correlations

Most modern predictors are "power predictors," designed to minimize empirical risk by using any available correlation. While effective for i.i.d. data, these models are fragile under agnostic distribution shift. Existing mitigation strategies often rely on reweighting samples to achieve feature independence. However, methods like Global Balancing Regression typically focus on first-order (pairwise) correlations and struggle with higher-order interactions. Furthermore, reweighting can be computationally prohibitive on large-scale datasets.

Methodology: From Experiments to Observational Data

The core innovation of BSSP is the application of Fractional Factorial Design (FFD)—a classic tool for planning controlled experiments—as a filtering mechanism for existing observational data.

1. Variable Deconfounding via FFD

In a Full Factorial Design, every possible combination of variables is tested to ensure orthogonality. Since this is infeasible for high-dimensional data, BSSP employs FFDs to select a subset of combinations that maintains independence up to a specific "resolution" (e.g., Resolution-5, which ensures independence between main effects and two-factor interactions).

2. The Subsampling Strategy

BSSP selects a subset of the training data that matches an FFD template. When a perfect match is unavailable due to data sparsity, the authors introduce a confounding measure ($\psi$) based on "generalized word-length patterns." This allows the algorithm to identify the "least-confounded" subset possible from the available observational pool.

Visual comparison of (a) Raw biased data, (b) Reweighting methods, and (c) the proposed BSSP Subsampling approach based on Fractional Factorial Design.

3. Model Agnosticism and Efficiency

Because the resulting subdata is balanced, any standard model (e.g., Linear or Logistic Regression) trained on it will naturally ignore confounded shortcuts. Additionally, because BSSP relies on subsampling rather than using the full dataset, the final training phase is often faster and requires fewer computational resources while achieving superior generalization.

Experimental Evaluation

The authors evaluated BSSP across three distinct environments:

  • Synthetic Simulations: In tasks where noisy features were intentionally correlated with outcomes via selection bias, BSSP estimated true causal coefficients with significantly higher accuracy than standard Ordinary Least Squares (OLS) or traditional reweighting methods.
  • WeChat Advertising Dataset: Predicting user "Likes" on advertisements. By training on one age group and testing on others with shifting behaviors, BSSP demonstrated superior stability, maintaining a low Root Mean Square Error (RMSE) where other models saw performance degradation.
  • Parkinson’s Telemonitoring: In a regression task using biomedical voice measurements, BSSP proved robust to distribution shifts caused by training on one patient group and testing on entirely different cohorts.

Discussion and Future Work

BSSP represents a significant step toward bridging experimental design and observational machine learning. By ensuring higher-order independence, it provides a more robust foundation for stable prediction than first-order balancing methods.

Current limitations include the reliance on discrete or dummy-encoded features. Future research aims to extend this framework to $k$-level designs and continuous variables using "space-filling designs." Despite these constraints, BSSP offers a powerful, efficient, and theoretically grounded solution for models operating in environments where the only constant is change.

Summary updated

Abstract

In machine learning, it is commonly assumed that training and test data share the same population distribution. However, this assumption is often violated in practice because the sample selection bias may induce the distribution shift from training data to test data. Such a model-agnostic distribution shift usually leads to prediction instability across unknown test data. In this paper, we propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design. It isolates the clear effect of each predictor from the confounding variables. A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift, hence improve both the accuracy of parameter estimation and prediction stability. Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
BibTeX
@misc{kuang2020balance,
  author = {Kun Kuang and Hengtao Zhang and Fei Wu and Yueting Zhuang and Aijun Zhang},
  title = {{Balance-Subsampled Stable Prediction}},
  year = {2020},
  eprint = {2006.04381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.04381},
}
Figure