Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets

Tianjian Li, Haoran Xu, Weiting Tan, Kenton Murray, Daniel Khashabi

Abstract

In training large-scale models on imbalanced datasets, researchers often treat upsampling data (Temperature Sampling) and upweighting loss (Scalarization) as mathematically equivalent. However, this equivalence breaks down under Stochastic Gradient Descent (SGD). We demonstrate that Temperature Sampling converges significantly faster due to lower gradient variance but is susceptible to overfitting. We propose COOLDOWN, a simple temperature-scheduling strategy that leverages the rapid convergence of upsampling while mitigating overfitting by transitioning to proportional sampling.

Introduction

The distribution of data in Large Language Models (LLMs) is inherently skewed. While high-resource languages (HRLs) like English are abundant, low-resource languages (LRLs) like Romanian or Swahili are scarce. Standard training objectives treat every token equally, often resulting in models that perform poorly on the long tail of the data distribution.

To address this, two primary balancing techniques are used:

  1. Scalarization (Upweighting): Maintaining the original data distribution but scaling the loss of rare examples.
  2. Temperature Sampling (Upsampling): Increasing the frequency of rare examples in the training batches.

While these methods are considered "mathematical twins," their practical impact on training dynamics under SGD differs significantly.

The Variance Gap: Sampling vs. Scalarization

The core finding of this work is that the choice between upsampling and upweighting is not merely a matter of implementation. Under SGD, these methods induce different gradient variances.

Theoretical Insights

We prove that while the two methods are equivalent under full gradient descent, they diverge under stochastic updates. Specifically, Temperature Sampling (TS) maintains a more stable gradient estimate compared to Scalarization (S). As the "temperature" $\tau$ increases—favoring rare data more aggressively—the variance in Scalarization grows exponentially, whereas it remains relatively stable for TS.

Gradient norm distribution for Sampling vs. Scalarization, showing significantly higher variance (std=2.25 vs 0.62) for the latter.

This "variance gap" explains why upweighting rare samples often leads to unstable training. It is akin to amplifying a faint signal along with its background noise, making it harder for the model to learn effectively.

The COOLDOWN Strategy

While upsampling provides a "speed hack" for convergence, it introduces an "overfitting trap." High-temperature sampling causes the model to see the same rare examples repeatedly, leading to memorization of noise rather than generalizable patterns.

To solve this, we introduce COOLDOWN. This strategy starts with a high temperature (e.g., $\tau=5$) to jumpstart learning on LRL data through low-variance updates. After a certain point, the temperature is "cooled down" to proportional sampling ($\tau=1$). This transition allows the model to benefit from early fast convergence without succumbing to late-stage overfitting.

Experimental Results

We evaluated COOLDOWN on Multilingual Machine Translation (OPUS-100, WMT) and Multilingual Language Modeling (mC4).

Convergence and Performance

COOLDOWN demonstrates superior performance by combining the strengths of both extremes. In machine translation tasks, it achieved a +3.1 BLEU improvement on LRLs compared to proportional sampling, with minimal impact on HRL performance.

Comparison of En-Ro Dev loss for different sampling strategies. COOLDOWN converges ~5x faster than proportional sampling and avoids the overfitting seen in constant high-temperature sampling.

As shown in the figure above, COOLDOWN matches the early convergence speed of high-temperature sampling (temp=5) but avoids the subsequent rise in validation loss (overfitting). It reaches optimal performance significantly faster than standard proportional sampling (temp=1).

Efficiency

Notably, COOLDOWN achieves these gains without the computational overhead of complex state-of-the-art methods like DoReMi, which require auxiliary proxy models. Furthermore, we found that simple "sparse" updates to the temperature schedule are more effective than continuous, "dense" adjustments.

Conclusion

The assumption that upsampling and upweighting are interchangeable is incorrect in the context of SGD. Upsampling's lower gradient variance makes it a powerful tool for accelerating training on imbalanced data. By using the COOLDOWN scheduling strategy, practitioners can harness this speed while maintaining model robustness against overfitting, providing a simple yet highly effective path for balanced multi-task and multilingual training.

Summary updated

BibTeX
@inproceedings{li2025upsample,
  author = {Tianjian Li and Haoran Xu and Weiting Tan and Kenton Murray and Daniel Khashabi},
  title = {{Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.171},
}
Figure