TL;DR
Researchers have introduced MiLe Loss, a new training objective that uses information entropy to help Large Language Models (LLMs) focus on "difficult-to-learn" tokens. By adaptively weighting the loss based on the model's uncertainty, MiLe Loss mitigates the inherent bias caused by the massive imbalance between frequent and infrequent words in natural language, leading to consistent performance gains across various benchmarks.
What Problem Does This Paper Address?
Natural language follows Zipf’s law: a handful of words (like "the," "is," "a") appear constantly, while the vast majority of the vocabulary appears rarely. When we train LLMs using standard Cross-Entropy (CE) loss, the model is overwhelmed by these frequent, "easy-to-learn" tokens. Consequently, it often neglects infrequent, "difficult-to-learn" tokens that carry significant information.
While techniques like Focal Loss exist to address class imbalance by down-weighting easy examples, the authors argue they aren't a perfect fit for language modeling. In object detection, there is usually one "right" answer. In language, however, many different tokens can be valid continuations of a sentence (e.g., "I like playing..." could be followed by "basketball," "football," or "games"). Focal Loss only looks at the probability of the specific ground-truth token, which can lead to misleading difficulty assessments in these "multi-label" scenarios.
Methods
The authors propose MiLe Loss (Entropy-Weighed Loss). Instead of looking only at the probability of the ground-truth token, MiLe Loss calculates the information entropy of the entire predicted probability distribution over the vocabulary.
- Low Entropy: The model is confident and its predictions are "focused" on a few tokens. This suggests the next token is easy to learn.
- High Entropy: The model is uncertain, and the probability distribution is more uniform. This suggests the token is difficult to learn.
MiLe Loss uses this entropy to dynamically scale the training loss. If a token is difficult (high entropy), the loss is scaled up, forcing the model to pay more attention. If it's easy (low entropy), the loss is scaled down. This approach naturally handles cases with multiple valid next tokens because a "focused" distribution on several valid options still results in relatively low entropy compared to total uncertainty.
Datasets
- Training: The models were pretrained on The Pile, a massive 825GB English corpus covering 22 domains (e.g., Wikipedia, GitHub, PubMed).
- Evaluation: The authors tested their models on several standard benchmarks:
- Common Sense Reasoning: BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, SIQA, StoryCloze, and Winogrande.
- Closed-Book QA: TriviaQA and WebQuestions.
- Multitask Understanding: MMLU (Massive Multitask Language Understanding).
Key Insights and Ideas
- Entropy as a Difficulty Proxy: The core "aha" moment is that entropy is a more robust measure of learning difficulty for language than single-token probability. It accounts for the inherent ambiguity of natural language.
- Mitigating Zipf's Bias: By explicitly addressing the token imbalance, the model becomes more "balanced" in its knowledge, rather than just being an expert at common stop-words.
- Hyperparameter Stability: The authors found that the scaling factor $\gamma$ (gamma) is relatively stable. While performance peaks at specific values (usually $\gamma=1$ or $\gamma=2$), MiLe Loss consistently outperforms standard Cross-Entropy across a range of settings.
Key Results
The researchers trained models at three scales: 468M, 1.2B, and 6.7B parameters.
- Consistent Gains: MiLe Loss steadily outperformed both Cross-Entropy and Focal Loss across almost all benchmarks and model sizes.
- Perplexity Reduction: A fine-grained analysis showed that while MiLe Loss slightly increases perplexity (a measure of how "surprised" the model is) for easy tokens, it significantly reduces perplexity for medium and difficult tokens. Since easy tokens dominate the dataset, the overall perplexity might rise slightly, but the model's performance on downstream tasks—which often rely on those difficult, informative tokens—improves.
- Scaling Benefits: The benefits of MiLe Loss actually increased as the models were trained on more tokens (moving from 100B to 200B tokens), suggesting the method scales well with longer training runs.
Limitations and Future Work
- Sensitivity to Noise: The authors acknowledge that MiLe Loss might be sensitive to low-quality or noisy training data. Noisy samples naturally result in high entropy, which could cause the model to over-index on "garbage" data by assigning it a higher loss weight.
- Domain Specificity: In the MMLU benchmark, MiLe Loss showed a slight performance decline in the "Social Sciences" field. The authors hypothesize that the model might be treating these samples as "easier" than they actually are, and they plan to investigate this domain-specific behavior in future work.
- Computational Cost: While not explicitly detailed as a major hurdle, calculating entropy over a 32k-sized vocabulary for every token adds a small amount of computational overhead compared to simple Cross-Entropy.
Summary updated