Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients

Jing An, Lexing Ying, Yuhua Zhu

In the world of machine learning, we frequently encounter "biased" data—datasets where the proportions of groups do not match the real world. If you train a model to detect a rare disease but 99% of your data consists of healthy patients, your model will likely just learn to predict "healthy" every time.

To correct this, practitioners typically reach for one of two tools: reweighting (giving more mathematical importance to rare samples in the loss function) or resampling (showing the model rare samples more frequently during training). On paper, these two methods are statistically equivalent. In practice, however, resampling almost always wins.

A paper from Stanford researchers, published at ICLR 2021, provides a theoretical explanation for this phenomenon: the performance gap stems entirely from how these methods interact with Stochastic Gradient Descent (SGD).

The Core Mystery: Equivalent in Theory, Divergent in Practice

The core mystery is the performance gap between two theoretically equivalent bias-correction techniques:

  • Reweighting multiplies the loss of each sample by a weight to compensate for its under-representation.
  • Resampling changes the frequency with which samples are seen during training (e.g., oversampling the minority group).

If you use a deterministic optimizer like full-batch Gradient Descent, these two methods yield identical results. However, modern deep learning almost exclusively relies on Stochastic Gradient Descent (SGD), which updates the model using small, random batches of data. The authors prove that the "stochastic" nature of SGD is exactly where the two methods diverge.

The Variance Trap and Optimization Trajectories

The "aha moment" of the paper is identifying the variance trap. Reweighting significantly increases the variance of stochastic gradients. When you assign a massive weight to a rare sample, a single batch containing that sample creates a massive "spike" in the gradient. This spike can violently kick the model out of a good region of the loss landscape.

Reweighting optimization trajectory

As seen in the reweighting trajectory above, the high variance causes the optimization path to bounce erratically, often escaping the optimal basin and getting trapped in a sub-optimal local minimum.

Resampling optimization trajectory

Conversely, resampling maintains a much smoother trajectory. Because it alters the frequency of data rather than the magnitude of the loss, it keeps gradient variance in check, allowing the model to smoothly converge to the correct minimum.

Theoretical Insights: Stability and SDEs

The authors use two primary mathematical frameworks to formalize this visual intuition:

  1. Dynamical Stability Analysis: By analyzing SGD as a discrete dynamical system, they derive stability conditions. They show that for reweighting to remain stable, you often need an impractically small learning rate. Furthermore, the stability of reweighting depends heavily on the original sampling proportions; if a group is very rare, reweighting becomes incredibly fragile. Resampling’s stability, however, is independent of how biased the original data was.
  2. Stochastic Differential Equations (SDEs): The researchers approximate the noisy path of SGD as a continuous-time process to calculate the stationary distribution—essentially, where the model is most likely to settle after a long training time. They prove that reweighting actually shifts the probability distribution of the model's final state. Even if the global minimum is at point A, the high noise of reweighting makes the model much more likely to settle in a sub-optimal local minimum at point B.

Empirical Validation Across Domains

The paper validates its theoretical findings using three distinct types of tasks, proving this is a general phenomenon:

  • Nonlinear Regression: On the California Housing Prices dataset with artificially introduced sampling bias, resampling achieved a significantly lower Mean Squared Error (MSE) and much lower variance across runs compared to reweighting.
  • Classification: On the highly skewed Bank Marketing dataset, oversampling the minority group led to the highest ROC-AUC scores (0.9516) compared to reweighting (0.9312) and the biased baseline (0.9277).
  • Off-policy Prediction: In a synthetic reinforcement learning environment testing Temporal Difference (TD) learning, the resampling approach stayed much closer to the true value function, while reweighting (Importance Sampling) deteriorated as the behavior policy became more biased.

The Takeaway: An Integrated View of Optimization

While the theoretical proofs currently rely on simpler geometries (like 1D examples or piecewise quadratic loss functions), the empirical results strongly suggest these dynamics hold true in complex deep learning landscapes.

The ultimate takeaway for practitioners is that we cannot treat the data, the model, and the optimizer as isolated components. The choice of objective function modification (like reweighting) fundamentally changes how the optimizer (SGD) perceives and navigates the loss landscape. When dealing with biased data, resampling offers a far more stable and robust path to convergence.

Summary updated

Abstract

A data set sampled from a certain population is biased if the subgroups of the population are sampled at proportions that are significantly different from their underlying proportions. Training machine learning models on biased data sets requires correction techniques to compensate for the bias. We consider two commonly-used techniques, resampling and reweighting, that rebalance the proportions of the subgroups to maintain the desired objective function. Though statistically equivalent, it has been observed that resampling outperforms reweighting when combined with stochastic gradient algorithms. By analyzing illustrative examples, we explain the reason behind this phenomenon using tools from dynamical stability and stochastic asymptotics. We also present experiments from regression, classification, and off-policy prediction to demonstrate that this is a general phenomenon. We argue that it is imperative to consider the objective function design and the optimization algorithm together while addressing the sampling bias.
BibTeX
@inproceedings{an2021why,
  author = {Jing An and Lexing Ying and Yuhua Zhu},
  title = {{Why resampling outperforms reweighting for correcting sampling bias with stochastic gradients}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=iQQK02mxVIT},
}
Figure