Abstract
In training large-scale models on imbalanced datasets, researchers often treat upsampling data (Temperature Sampling) and upweighting loss (Scalarization) as mathematically equivalent. However, this equivalence breaks down under Stochastic Gradient Descent (SGD). We demonstrate that Temperature Sampling converges significantly faster due to lower gradient variance but is susceptible to overfitting. We propose COOLDOWN, a simple temperature-scheduling strategy that leverages the rapid convergence of upsampling while mitigating overfitting by transitioning to proportional sampling.
Introduction
The distribution of data in Large Language Models (LLMs) is inherently skewed. While high-resource languages (HRLs) like English are abundant, low-resource languages (LRLs) like Romanian or Swahili are scarce. Standard training objectives treat every token equally, often resulting in models that perform poorly on the long tail of the data distribution.
To address this, two primary balancing techniques are used:
- Scalarization (Upweighting): Maintaining the original data distribution but scaling the loss of rare examples.
- Temperature Sampling (Upsampling): Increasing the frequency of rare examples in the training batches.
While these methods are considered "mathematical twins," their practical impact on training dynamics under SGD differs significantly.
The Variance Gap: Sampling vs. Scalarization
The core finding of this work is that the choice between upsampling and upweighting is not merely a matter of implementation. Under SGD, these methods induce different gradient variances.
Theoretical Insights
We prove that while the two methods are equivalent under full gradient descent, they diverge under stochastic updates. Specifically, Temperature Sampling (TS) maintains a more stable gradient estimate compared to Scalarization (S). As the "temperature" $\tau$ increases—favoring rare data more aggressively—the variance in Scalarization grows exponentially, whereas it remains relatively stable for TS.
This "variance gap" explains why upweighting rare samples often leads to unstable training. It is akin to amplifying a faint signal along with its background noise, making it harder for the model to learn effectively.
The COOLDOWN Strategy
While upsampling provides a "speed hack" for convergence, it introduces an "overfitting trap." High-temperature sampling causes the model to see the same rare examples repeatedly, leading to memorization of noise rather than generalizable patterns.
To solve this, we introduce COOLDOWN. This strategy starts with a high temperature (e.g., $\tau=5$) to jumpstart learning on LRL data through low-variance updates. After a certain point, the temperature is "cooled down" to proportional sampling ($\tau=1$). This transition allows the model to benefit from early fast convergence without succumbing to late-stage overfitting.
Experimental Results
We evaluated COOLDOWN on Multilingual Machine Translation (OPUS-100, WMT) and Multilingual Language Modeling (mC4).
Convergence and Performance
COOLDOWN demonstrates superior performance by combining the strengths of both extremes. In machine translation tasks, it achieved a +3.1 BLEU improvement on LRLs compared to proportional sampling, with minimal impact on HRL performance.
As shown in the figure above, COOLDOWN matches the early convergence speed of high-temperature sampling (temp=5) but avoids the subsequent rise in validation loss (overfitting). It reaches optimal performance significantly faster than standard proportional sampling (temp=1).
Efficiency
Notably, COOLDOWN achieves these gains without the computational overhead of complex state-of-the-art methods like DoReMi, which require auxiliary proxy models. Furthermore, we found that simple "sparse" updates to the temperature schedule are more effective than continuous, "dense" adjustments.
Conclusion
The assumption that upsampling and upweighting are interchangeable is incorrect in the context of SGD. Upsampling's lower gradient variance makes it a powerful tool for accelerating training on imbalanced data. By using the COOLDOWN scheduling strategy, practitioners can harness this speed while maintaining model robustness against overfitting, providing a simple yet highly effective path for balanced multi-task and multilingual training.
Summary updated