Why resampling outperforms reweighting for correcting sampling bias

Jing An, Lexing Ying, Yuhua Zhu

Abstract

A data set sampled from a certain population is biased if the subgroups of the population are sampled at proportions that are significantly different from their underlying proportions. Training machine learning models on biased data sets requires correction techniques to compensate for the bias. We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function. Though statistically equivalent, it has been observed that resampling outperforms reweighting when combined with stochastic gradient algorithms. By analyzing illustrative examples, we explain the reason behind this phenomenon using tools from dynamical stability and stochastic asymptotics. We also present experiments from regression, classification, and off-policy prediction to demonstrate that this is a general phenomenon. We argue that it is imperative to consider the objective function design and the optimization algorithm together while addressing the sampling bias.

Introduction

In modern machine learning, datasets are rarely perfect reflections of the underlying population. Sampling bias—where certain subgroups are overrepresented or underrepresented—is a pervasive challenge. To address this, practitioners typically employ one of two strategies: reweighting, which keeps the data as-is but scales the loss of minority samples, or resampling, which physically alters the dataset through oversampling or undersampling.

While these methods are mathematically equivalent in terms of their expected loss functions, a "folk theorem" has long persisted in the community: resampling consistently yields better performance when training with Stochastic Gradient Descent (SGD). This paper provides a theoretical and quantitative analysis of this phenomenon, demonstrating that the choice of bias correction technique fundamentally alters the optimization landscape and the stability of the training process.

Theoretical Framework: Stability and Noise

The authors analyze the interaction between bias correction and optimization through two primary lenses: discrete dynamical stability and continuous-time Stochastic Differential Equations (SDEs). By treating SGD as a dynamical system, they examine the stability of stationary points (minima).

Comparison of optimization trajectories on a double-well potential. Reweighting (left) leads to unstable jumps between minima, whereas resampling (right) remains stable.

A critical finding is the Variance Trap. In reweighting, assigning a large weight to a rare subgroup causes the gradients calculated from those samples to become massive. This introduces high-variance noise into the optimization process. While resampling also increases the frequency of these gradients, it does not scale their magnitude, leading to a more stable noise landscape. This difference is crucial; as sampling bias increases, reweighting requires a significantly smaller learning rate to remain stable, whereas resampling remains robust regardless of the sampling proportions.

The Escape Phenomenon

Using SDE theory, the paper illustrates the "Escape Phenomenon." Even when a loss function possesses a clear global minimum, the unbalanced noise generated by reweighting can cause the model parameters to "evaporate" out of a good minimum and settle into a suboptimal local one. This suggests that the noise landscape is just as important as the loss landscape itself. The authors argue that sampling bias correction must be viewed as an integrated system problem where the data, the model, and the optimizer are inextricably linked.

Empirical Results

The theoretical predictions were validated across synthetic toy examples and real-world datasets in classification, regression, and reinforcement learning.

Long-term iteration dynamics of reweighting vs. resampling. While reweighting (middle) eventually escapes the local minimum due to high gradient variance, resampling (right) maintains a stable stationary distribution.
  • Classification: On the Bank Marketing dataset (highly skewed classes), resampling achieved a significantly higher ROC-AUC (0.9516) compared to reweighting (0.9312).
  • Regression: In the California Housing task, resampling resulted in lower Mean Squared Error (7.96e+04) than reweighting (9.04e+04 to 9.35e+04) and demonstrated much lower variance across different training runs.
  • Reinforcement Learning: In off-policy prediction, resampling maintained proximity to the exact value function, while reweighting performance deteriorated as the discrepancy between behavior and target policies increased.

Conclusion and Future Work

This work demonstrates that resampling is superior to reweighting for correcting sampling bias because it maintains stable gradient variance during SGD. Future research should extend this analysis to high-dimensional, non-convex landscapes typical of deep neural networks and explore its implications for unsupervised learning tasks like PCA on biased subgroups. Ultimately, practitioners should favor resampling techniques to ensure numerical stability and better convergence to global optima.

Summary updated

Abstract

A data set sampled from a certain population is biased if the subgroups of the population are sampled at proportions that are significantly different from their underlying proportions. Training machine learning models on biased data sets requires correction techniques to compensate for the bias. We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function. Though statistically equivalent, it has been observed that resampling outperforms reweighting when combined with stochastic gradient algorithms. By analyzing illustrative examples, we explain the reason behind this phenomenon using tools from dynamical stability and stochastic asymptotics. We also present experiments from regression, classification, and off-policy prediction to demonstrate that this is a general phenomenon. We argue that it is imperative to consider the objective function design and the optimization algorithm together while addressing the sampling bias.
BibTeX
@inproceedings{an2021why,
  author = {Jing An and Lexing Ying and Yuhua Zhu},
  title = {{Why resampling outperforms reweighting for correcting sampling bias}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=iQQK02mxVIT},
}
Figure