Learning Is Not A Race: Improving Retrieval in Language Models via Equal Learning

Wanqian Yang, Aahlad Manas Puli, Rajesh RanganathEMNLP 2025

Abstract

Large Language Models (LLMs) often struggle with simple retrieval tasks despite achieving near-zero training loss. This paper identifies "unequal learning" as a primary cause, where models overfit easy shortcuts in the data, neglecting generalizable features. We introduce Equal Learning (E-LEARN), a framework that balances the learning rate across all tokens using specialized loss functions and learning rate schedules. Our results demonstrate significant improvements in retrieval performance across synthetic and natural language benchmarks.

Introduction: The Retrieval Paradox

We often assume that if an LLM achieves near-zero training loss, it has successfully "learned" the task. However, for retrieval—the ability to extract specific information from a provided context—this is not always true. Models frequently take shortcuts, identifying simple mechanisms to predict some tokens early in training. Once these "easy" tokens provide no further gradient signal, the model uses its remaining capacity to memorize the rest of the sequence as noise. This phenomenon, termed unequal learning, leads to models that appear perfect on training data but fail when context lengths increase or shortcuts are removed.

Key Insights: Stratification and Shortcuts

The core issue lies in how models prioritize information. Training often begins with stratification, where easy tokens are fitted first. If a model does not rectify—that is, learn the correct underlying mechanism—before easy tokens reach zero loss, it resorts to memorization.

Accuracy degradation on hard samples as context length increases.

As shown in the figure above, while training accuracy remains high, test accuracy on "hard" samples (those without shortcuts) plummets as context length increases. This suggests that longer contexts provide more "noisy" features that models exploit as shortcuts to minimize training loss without learning the actual task.

Methods: The E-LEARN Framework

To combat unequal learning, the authors propose E-LEARN, shifting the objective from rapid loss minimization to balanced learning across all data parts.

  1. Explicit Regularization (E-LOSSES): Standard cross-entropy encourages infinite margins for correct tokens. E-LEARN introduces losses (E1, E2, E3) that penalize large or high-variance margins, forcing a uniform margin across all tokens and preventing "easy" tokens from being solved too quickly.
  2. Learning Rate Warmup: A small initial learning rate prevents the model from "sprinting" toward suboptimal local minima. This ensures all tokens contribute to the gradient signal longer, facilitating the discovery of generalizable mechanisms.
Training loss disparity between easy and hard samples, illustrating stratification.

The training dynamics often show a stark contrast: easy samples reach near-zero loss almost immediately, while hard samples require significantly more training steps to converge. E-LEARN aims to bridge this gap by ensuring the model doesn't stop learning the hard features once the easy ones are mastered.

Experimental Results

The researchers evaluated E-LEARN on several benchmarks:

  • Padded-Shifted Copy Task: Standard transformers failed as context length grew, while E-LEARN maintained near-perfect accuracy.
  • Graph Navigation: On the $G_{5,3}$ graph task, E1 loss improved accuracy from 0.001 (baseline) to 0.117.
  • Natural Language Fine-Tuning: Fine-tuning GPT-2 with E-LEARN improved Shakespearean text retrieval from 0.703 to 0.766 and King James Bible retrieval from 0.863 to 0.981.

Conclusion

E-LEARN demonstrates that "slow and steady" training can lead to better generalization in retrieval tasks. By preventing models from wasting capacity on shortcuts, the framework ensures that the "quack" of low training loss actually represents a "duck" of true learning. Future work will explore the impact of equal learning on more complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{yang2025learning,
  author = {Wanqian Yang and Aahlad Manas Puli and Rajesh Ranganath},
  title = {{Learning Is Not A Race: Improving Retrieval in Language Models via Equal Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure