Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets

Tianjian Li, Haoran Xu, Weiting Tan, Kenton Murray, Daniel Khashabi

Introduction

In the era of massive datasets, researchers frequently encounter the "long-tail" problem. In multilingual modeling, for instance, high-resource languages like English may outweigh low-resource languages like Swahili by a factor of 2,000 to 1. Training on such raw mixtures often leads to models that excel in dominant languages while failing significantly on minority ones. To address this imbalance, practitioners typically employ two strategies: Scalarization (Upweighting), which multiplies the loss of rare samples by a constant, and Temperature Sampling (Upsampling), which increases the frequency of rare data points during training.

While these methods are often considered mathematically equivalent in theory, this paper demonstrates that they behave quite differently in practice when using Stochastic Gradient Descent (SGD). The authors introduce COOLDOWN, a simple scheduling strategy that leverages the strengths of both approaches to achieve faster convergence and better generalization.

The Variance Problem

The core insight of the paper is that the mathematical equivalence between upweighting and upsampling holds only under full gradient descent. In the context of mini-batch SGD, Scalarization induces significantly higher variance in gradient estimation. By scaling the loss of a single sample, Scalarization allows outliers or noisy samples to exert a disproportionate influence on the gradient, leading to erratic optimization steps.

Empirical comparison of gradient norm variance between sampling and scalarization.

Empirical measurements of gradient norms confirm this volatility. As shown in Figure 2, the standard deviation and maximum gradient norms for Scalarization are substantially higher than those for Temperature Sampling. Upsampling effectively averages noise across multiple instances of the data, resulting in a smoother and more stable optimization path.

The COOLDOWN Strategy

Building on the observation that high-temperature upsampling accelerates early learning but eventually leads to overfitting on low-resource data, the authors propose COOLDOWN. This is a two-stage scheduling strategy designed to optimize both speed and final performance:

  1. Stage 1 (Warm-up): Utilize a high sampling temperature (e.g., $\tau=5$) to aggressively upsample low-resource data, acting as a "turbo button" for initial convergence.
  2. Stage 2 (Cool-down): Reduce the temperature to $\tau=1$ (proportional sampling) to stabilize the model and prevent it from memorizing the limited low-resource datasets.
COOLDOWN performance compared to fixed temperature sampling strategies.

As illustrated in Figure 1, COOLDOWN converges approximately 5x faster than standard proportional sampling (temp=1) while avoiding the performance degradation and overfitting seen with persistent high-temperature upsampling (temp=5).

Experimental Results

The authors evaluated COOLDOWN across Multilingual Machine Translation (MMT) using WMT and OPUS-100 datasets, and Multilingual Language Modeling (MLM) using the mC4 corpus.

Key findings include:

  • Superior Translation Performance: On the OPUS-100 benchmark, COOLDOWN outperformed proportional sampling by +3.1 BLEU on low-resource languages without sacrificing performance on high-resource ones.
  • Efficiency and Simplicity: COOLDOWN matched the performance of more complex state-of-the-art methods like DoReMi, which requires training auxiliary "proxy models." In contrast, COOLDOWN is trivial to implement and adds no computational overhead.
  • Optimal Scheduling: Experiments showed that "sparse" updates—a single significant drop in temperature—performed better than continuous "dense" updates. This is likely because dense updates often rely on Scalarization-based approximations that reintroduce high gradient variance.

Conclusion

This research clarifies a long-standing assumption in machine learning by proving that upsampling is superior to upweighting due to its variance-reducing properties. The proposed COOLDOWN strategy offers a practical, efficient way to train on imbalanced datasets, providing a significant boost in convergence speed and final model quality. Future work may explore adaptive triggers for the "cool-down" phase based on real-time validation metrics.

Summary updated

Abstract

Data abundance across different domains exhibits a long-tailed distribution: few domains have abundant data, while most face data scarcity. Our work focuses on a multilingual setting, where available data is heavily skewed towards high-resource languages. Two common strategies to address this disparity are upsampling low-resource data (Temperature Sampling) and upweighting low-resource loss (Scalarization). These methods are often assumed to be equivalent, but this equivalence has not been rigorously established, prompting our investigation. Through theoretical and empirical analysis, we identify when these two methods are equivalent and when they diverge. We prove that they are equivalent under full gradient descent but differ under stochastic gradient descent due to differences in gradient variance. Specifically, Temperature Sampling exhibits lower variance in gradient estimation compared to Scalarization, leading to faster convergence but a higher risk of overfitting. Based on these insights, we propose Cooldown, a strategy that starts by heavily upsampling low-resource languages to accelerate convergence and gradually reduces the upsampling to prevent overfitting -- achieving the best of both worlds. Our method competes effectively with existing data re-weighting techniques while offering computational efficiency.
BibTeX
@inproceedings{li2025upsample,
  author = {Tianjian Li and Haoran Xu and Weiting Tan and Kenton Murray and Daniel Khashabi},
  title = {{Upsample or Upweight? Balanced Training on Heavily Imbalanced Datasets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.171},
}
Figure