Abstract
Machine learning models often fail under distribution shift due to their tendency to exploit spurious correlations present in training data. We present Balance-Subsampled Stable Prediction (BSSP), a novel framework that utilizes Fractional Factorial Design (FFD) to subsample observational data into a balanced, deconfounded subset. By ensuring feature independence within the training subdata, BSSP forces models to focus on stable, causal relationships rather than selection-biased shortcuts. Experimental results on synthetic and real-world datasets, including advertising and biomedical telemonitoring, demonstrate that BSSP significantly improves prediction stability and accuracy across unknown test distributions.
Introduction
A fundamental assumption in standard machine learning is that training and test data are independent and identically distributed (i.e., the i.i.d. assumption). However, in real-world applications, sample selection bias often creates spurious correlations—relationships that exist in the training set but vanish or reverse in the test set. For instance, a model trained to recognize dogs primarily in grassy environments may fail when encountering a dog on a beach, having incorrectly identified "grass" as a predictive feature for "dog."
To address this, the BSSP framework treats observational data through the lens of experimental statistics, specifically borrowing from the theory of Fractional Factorial Design to eliminate confounding factors at the source.
The Challenge of Spurious Correlations
Most modern predictors are "power predictors," designed to minimize empirical risk by using any available correlation. While effective for i.i.d. data, these models are fragile under agnostic distribution shift. Existing mitigation strategies often rely on reweighting samples to achieve feature independence. However, methods like Global Balancing Regression typically focus on first-order (pairwise) correlations and struggle with higher-order interactions. Furthermore, reweighting can be computationally prohibitive on large-scale datasets.
Methodology: From Experiments to Observational Data
The core innovation of BSSP is the application of Fractional Factorial Design (FFD)—a classic tool for planning controlled experiments—as a filtering mechanism for existing observational data.
1. Variable Deconfounding via FFD
In a Full Factorial Design, every possible combination of variables is tested to ensure orthogonality. Since this is infeasible for high-dimensional data, BSSP employs FFDs to select a subset of combinations that maintains independence up to a specific "resolution" (e.g., Resolution-5, which ensures independence between main effects and two-factor interactions).
2. The Subsampling Strategy
BSSP selects a subset of the training data that matches an FFD template. When a perfect match is unavailable due to data sparsity, the authors introduce a confounding measure ($\psi$) based on "generalized word-length patterns." This allows the algorithm to identify the "least-confounded" subset possible from the available observational pool.
3. Model Agnosticism and Efficiency
Because the resulting subdata is balanced, any standard model (e.g., Linear or Logistic Regression) trained on it will naturally ignore confounded shortcuts. Additionally, because BSSP relies on subsampling rather than using the full dataset, the final training phase is often faster and requires fewer computational resources while achieving superior generalization.
Experimental Evaluation
The authors evaluated BSSP across three distinct environments:
- Synthetic Simulations: In tasks where noisy features were intentionally correlated with outcomes via selection bias, BSSP estimated true causal coefficients with significantly higher accuracy than standard Ordinary Least Squares (OLS) or traditional reweighting methods.
- WeChat Advertising Dataset: Predicting user "Likes" on advertisements. By training on one age group and testing on others with shifting behaviors, BSSP demonstrated superior stability, maintaining a low Root Mean Square Error (RMSE) where other models saw performance degradation.
- Parkinson’s Telemonitoring: In a regression task using biomedical voice measurements, BSSP proved robust to distribution shifts caused by training on one patient group and testing on entirely different cohorts.
Discussion and Future Work
BSSP represents a significant step toward bridging experimental design and observational machine learning. By ensuring higher-order independence, it provides a more robust foundation for stable prediction than first-order balancing methods.
Current limitations include the reliance on discrete or dummy-encoded features. Future research aims to extend this framework to $k$-level designs and continuous variables using "space-filling designs." Despite these constraints, BSSP offers a powerful, efficient, and theoretically grounded solution for models operating in environments where the only constant is change.
Summary updated