MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models

Zhenpeng Su, Xing Wu, Xue Bai, Zijia Lin, Hui Chen, Guiguang Ding, Wei Zhou, Songlin Hu

Abstract

Generative language models are usually pretrained on large text corpus via predicting the next token given the previous ones. Recent works have demonstrated the impressive performance of large generative language models on downstream tasks. However, existing generative language models generally neglect an inherent challenge in text corpus during training: the imbalance between frequent tokens and infrequent ones. This can lead a language model to be dominated by common and easy-to-learn tokens, thereby overlooking the infrequent and difficult-to-learn ones. To alleviate this, we propose MiLe Loss for mitigating the bias of learning difficulties with tokens. During training, it dynamically assesses the learning difficulty of a token based on the information entropy of the predicted probability distribution. Experiments on the Pile dataset with models up to 6.7B parameters reveal consistent performance improvements on downstream benchmarks.

Introduction: The Challenge of Imbalanced Language Data

Language is inherently imbalanced, following Zipf’s Law where a few frequent tokens (e.g., "the", "is", "a") dominate any given corpus. Standard training typically employs Cross-Entropy (CE) loss, which treats every token prediction as equally important. Consequently, models become highly optimized for these "easy" cases at the expense of "difficult," information-rich tokens that appear less frequently.

While techniques like Focal Loss attempt to address this in computer vision by weighting loss based on ground-truth probability, they often fail in language modeling. This is due to the "multi-label" nature of language: given a prompt like "I like playing...", multiple completions such as "basketball," "football," or "golf" are all valid. Focal Loss would unfairly penalize a model for being uncertain even if it correctly identifies several plausible options.

Methodology: Entropy-Weighted Learning

The proposed MiLe Loss (Mitigating Learning difficulties Loss) replaces simple probability-based weighting with a dynamic weight based on Information Entropy.

Visualization of token prediction probabilities and entropy-based difficulty.

Entropy as a Difficulty Metric

During training, the model generates a probability distribution over the entire vocabulary for the next token. A high-entropy distribution indicates model uncertainty or inherent difficulty in predicting the next token. MiLe Loss scales the standard cross-entropy loss by a factor derived from this entropy:

$$\mathcal{L}_{IL} = -(1 + H(p))^\gamma \log(p_{t_i})$$

where $H(p)$ is the entropy of the predicted distribution and $\gamma$ is a hyperparameter controlling the focus on difficult tokens. Low-entropy (easy) tokens receive reduced weight, while high-entropy (hard) tokens receive increased weight, forcing the model to focus more on difficult-to-learn information.

Experimental Setup

Models were trained on The Pile, a massive 825GB English text corpus, at three scales: 468M, 1.2B, and 6.7B parameters. Evaluation spanned common-sense reasoning (e.g., BoolQ, HellaSwag), closed-book QA (TriviaQA), and the MMLU benchmark covering 57 subjects.

Key Results and Performance Gains

MiLe Loss consistently outperformed both Cross-Entropy and Focal Loss across nearly all benchmarks.

Average accuracy across different model scales (468M, 1.2B, 6.7B) as a function of the scaling hyperparameter γ.
  • Scaling Benefits: The performance gap between MiLe Loss and standard training increased as models were trained on more tokens (up to 200B), suggesting the method is highly robust for long-term training.
  • Zero-Shot Success: On the TriviaQA benchmark, the MiLe-trained 6.7B model achieved a 20.64% exact match score, significantly higher than the 17.09% baseline.
  • The Perplexity Paradox: Interestingly, models trained with MiLe Loss sometimes showed slightly higher validation perplexity than standard models while performing better on downstream tasks. This suggests standard perplexity is overly influenced by "easy" tokens, which do not necessarily correlate with reasoning or knowledge acquisition.

Discussion and Limitations

While effective, MiLe Loss may be sensitive to noise; "garbage" text or outliers naturally result in high entropy, potentially causing the model to over-weight irrelevant data. Future work will focus on filtering high-entropy noise to ensure the model focuses on meaningful semantic difficulty. Additionally, the $\gamma$ parameter requires careful tuning; if set too high, the model may ignore easy tokens entirely, while a low value reverts to standard training behavior.

Summary updated

Abstract

Generative language models are usually pretrained on large text corpus via predicting the next token (i.e., sub-word/word/phrase) given the previous ones. Recent works have demonstrated the impressive performance of large generative language models on downstream tasks. However, existing generative language models generally neglect an inherent challenge in text corpus during training, i.e., the imbalance between frequent tokens and infrequent ones. It can lead a language model to be dominated by common and easy-to-learn tokens, thereby overlooking the infrequent and difficult-to-learn ones. To alleviate that, we propose a MiLe Loss function for mitigating the bias of learning difficulties with tokens. During training, it can dynamically assess the learning difficulty of a to-be-learned token, according to the information entropy of the corresponding predicted probability distribution over the vocabulary. Then it scales the training loss adaptively, trying to lead the model to focus more on the difficult-to-learn tokens. On the Pile dataset, we train generative language models at different scales of 468M, 1.2B, and 6.7B parameters. Experiments reveal that models incorporating the proposed MiLe Loss can gain consistent performance improvement on downstream benchmarks.
BibTeX
@misc{su2023mile,
  author = {Zhenpeng Su and Xing Wu and Xue Bai and Zijia Lin and Hui Chen and Guiguang Ding and Wei Zhou and Songlin Hu},
  title = {{MiLe Loss: a New Entropy-Weighed Loss for Mitigating the Bias of Learning Difficulties in Large Language Models}},
  year = {2023},
  eprint = {2310.19531},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.19531},
}
Figure