Introduction
In the era of massive datasets, researchers frequently encounter the "long-tail" problem. In multilingual modeling, for instance, high-resource languages like English may outweigh low-resource languages like Swahili by a factor of 2,000 to 1. Training on such raw mixtures often leads to models that excel in dominant languages while failing significantly on minority ones. To address this imbalance, practitioners typically employ two strategies: Scalarization (Upweighting), which multiplies the loss of rare samples by a constant, and Temperature Sampling (Upsampling), which increases the frequency of rare data points during training.
While these methods are often considered mathematically equivalent in theory, this paper demonstrates that they behave quite differently in practice when using Stochastic Gradient Descent (SGD). The authors introduce COOLDOWN, a simple scheduling strategy that leverages the strengths of both approaches to achieve faster convergence and better generalization.
The Variance Problem
The core insight of the paper is that the mathematical equivalence between upweighting and upsampling holds only under full gradient descent. In the context of mini-batch SGD, Scalarization induces significantly higher variance in gradient estimation. By scaling the loss of a single sample, Scalarization allows outliers or noisy samples to exert a disproportionate influence on the gradient, leading to erratic optimization steps.
Empirical measurements of gradient norms confirm this volatility. As shown in Figure 2, the standard deviation and maximum gradient norms for Scalarization are substantially higher than those for Temperature Sampling. Upsampling effectively averages noise across multiple instances of the data, resulting in a smoother and more stable optimization path.
The COOLDOWN Strategy
Building on the observation that high-temperature upsampling accelerates early learning but eventually leads to overfitting on low-resource data, the authors propose COOLDOWN. This is a two-stage scheduling strategy designed to optimize both speed and final performance:
- Stage 1 (Warm-up): Utilize a high sampling temperature (e.g., $\tau=5$) to aggressively upsample low-resource data, acting as a "turbo button" for initial convergence.
- Stage 2 (Cool-down): Reduce the temperature to $\tau=1$ (proportional sampling) to stabilize the model and prevent it from memorizing the limited low-resource datasets.
As illustrated in Figure 1, COOLDOWN converges approximately 5x faster than standard proportional sampling (temp=1) while avoiding the performance degradation and overfitting seen with persistent high-temperature upsampling (temp=5).
Experimental Results
The authors evaluated COOLDOWN across Multilingual Machine Translation (MMT) using WMT and OPUS-100 datasets, and Multilingual Language Modeling (MLM) using the mC4 corpus.
Key findings include:
- Superior Translation Performance: On the OPUS-100 benchmark, COOLDOWN outperformed proportional sampling by +3.1 BLEU on low-resource languages without sacrificing performance on high-resource ones.
- Efficiency and Simplicity: COOLDOWN matched the performance of more complex state-of-the-art methods like DoReMi, which requires training auxiliary "proxy models." In contrast, COOLDOWN is trivial to implement and adds no computational overhead.
- Optimal Scheduling: Experiments showed that "sparse" updates—a single significant drop in temperature—performed better than continuous "dense" updates. This is likely because dense updates often rely on Scalarization-based approximations that reintroduce high gradient variance.
Conclusion
This research clarifies a long-standing assumption in machine learning by proving that upsampling is superior to upweighting due to its variance-reducing properties. The proposed COOLDOWN strategy offers a practical, efficient way to train on imbalanced datasets, providing a significant boost in convergence speed and final model quality. Future work may explore adaptive triggers for the "cool-down" phase based on real-time validation metrics.
Summary updated