Abstract
Generative language models are usually pretrained on large text corpus via predicting the next token given the previous ones. Recent works have demonstrated the impressive performance of large generative language models on downstream tasks. However, existing generative language models generally neglect an inherent challenge in text corpus during training: the imbalance between frequent tokens and infrequent ones. This can lead a language model to be dominated by common and easy-to-learn tokens, thereby overlooking the infrequent and difficult-to-learn ones. To alleviate this, we propose MiLe Loss for mitigating the bias of learning difficulties with tokens. During training, it dynamically assesses the learning difficulty of a token based on the information entropy of the predicted probability distribution. Experiments on the Pile dataset with models up to 6.7B parameters reveal consistent performance improvements on downstream benchmarks.
Introduction: The Challenge of Imbalanced Language Data
Language is inherently imbalanced, following Zipf’s Law where a few frequent tokens (e.g., "the", "is", "a") dominate any given corpus. Standard training typically employs Cross-Entropy (CE) loss, which treats every token prediction as equally important. Consequently, models become highly optimized for these "easy" cases at the expense of "difficult," information-rich tokens that appear less frequently.
While techniques like Focal Loss attempt to address this in computer vision by weighting loss based on ground-truth probability, they often fail in language modeling. This is due to the "multi-label" nature of language: given a prompt like "I like playing...", multiple completions such as "basketball," "football," or "golf" are all valid. Focal Loss would unfairly penalize a model for being uncertain even if it correctly identifies several plausible options.
Methodology: Entropy-Weighted Learning
The proposed MiLe Loss (Mitigating Learning difficulties Loss) replaces simple probability-based weighting with a dynamic weight based on Information Entropy.
Entropy as a Difficulty Metric
During training, the model generates a probability distribution over the entire vocabulary for the next token. A high-entropy distribution indicates model uncertainty or inherent difficulty in predicting the next token. MiLe Loss scales the standard cross-entropy loss by a factor derived from this entropy:
$$\mathcal{L}_{IL} = -(1 + H(p))^\gamma \log(p_{t_i})$$
where $H(p)$ is the entropy of the predicted distribution and $\gamma$ is a hyperparameter controlling the focus on difficult tokens. Low-entropy (easy) tokens receive reduced weight, while high-entropy (hard) tokens receive increased weight, forcing the model to focus more on difficult-to-learn information.
Experimental Setup
Models were trained on The Pile, a massive 825GB English text corpus, at three scales: 468M, 1.2B, and 6.7B parameters. Evaluation spanned common-sense reasoning (e.g., BoolQ, HellaSwag), closed-book QA (TriviaQA), and the MMLU benchmark covering 57 subjects.
Key Results and Performance Gains
MiLe Loss consistently outperformed both Cross-Entropy and Focal Loss across nearly all benchmarks.
- Scaling Benefits: The performance gap between MiLe Loss and standard training increased as models were trained on more tokens (up to 200B), suggesting the method is highly robust for long-term training.
- Zero-Shot Success: On the TriviaQA benchmark, the MiLe-trained 6.7B model achieved a 20.64% exact match score, significantly higher than the 17.09% baseline.
- The Perplexity Paradox: Interestingly, models trained with MiLe Loss sometimes showed slightly higher validation perplexity than standard models while performing better on downstream tasks. This suggests standard perplexity is overly influenced by "easy" tokens, which do not necessarily correlate with reasoning or knowledge acquisition.
Discussion and Limitations
While effective, MiLe Loss may be sensitive to noise; "garbage" text or outliers naturally result in high entropy, potentially causing the model to over-weight irrelevant data. Future work will focus on filtering high-entropy noise to ensure the model focuses on meaningful semantic difficulty. Additionally, the $\gamma$ parameter requires careful tuning; if set too high, the model may ignore easy tokens entirely, while a low value reverts to standard training behavior.
Summary updated