Abstract
Language models (LMs) store diverse factual knowledge in their parameters during self-supervised training on unlabeled documents. While instruction-tuning makes this knowledge extractable, LMs often suffer from the "perplexity curse": despite minimizing document perplexity during training, they struggle to extract information via question prompts. This study investigates this phenomenon by fine-tuning LMs on new data, revealing a significant positional bias. Models struggle to answer questions about information located in the middle or at the end of training documents. We demonstrate that this issue stems from standard auto-regressive training and propose that adding regularization, specifically Denoising Auto-Regressive (D-AR) training, mitigates the bias. Our findings provide a key to more effectively extracting parametric knowledge from LMs.
The Perplexity Curse and Positional Bias
A common assumption in LLM training is that low perplexity—the ability to predict the next token with high confidence—equates to successful knowledge acquisition. However, researchers have identified a "perplexity curse" where a model can recite a training document verbatim but fail to answer simple questions about facts contained within it.
This failure is deeply tied to the position of the information. As shown in Figure 1, if a fact is moved from the beginning of a document to the end during training, the model's ability to retrieve that specific knowledge at inference time drops significantly. This suggests that auto-regressive models become overly dependent on specific token prefixes. When an inference prompt (like a question) does not match the specific prefix seen during training, the model fails to retrieve the "parametric knowledge" stored in its weights.
Regularization Strategies for Knowledge Extraction
To break the rigid dependency on specific prefixes and improve knowledge extraction across all document positions, the authors investigated four training strategies:
- Standard Auto-Regressive (AR): The baseline method of next-token prediction.
- Denoising Auto-Regressive (D-AR): The primary proposal. During training, $R\%$ of input tokens are replaced with random tokens. Crucially, the model must still predict the original next token. This forces the model to rely on broader factual connections rather than specific local sequences.
- Sentence Shuffling: Randomizing the order of sentences within a document to prevent the model from learning a fixed global position for facts.
- Attention Dropout: Randomly dropping elements of the attention mask to reduce the model's reliance on specific preceding tokens.
Experimental Setup and Datasets
The study utilized three distinct datasets to isolate positional effects:
- SynthLang: A synthetic environment where models learn five languages spoken by different individuals, allowing for controlled testing of recall based on the order of mention.
- Wiki2023+: Wikipedia articles published in 2023 (post-dating the models' original pre-training) with annotated QA pairs linked to specific sentences.
- SynthID: A high-difficulty task requiring the memorization of ten unique 10-character identification numbers for different individuals.
Key Results and Analysis
The experiments yielded several critical insights into how LLMs store and retrieve information:
- Significant Accuracy Gains: On the Wiki2023+ dataset, a Llama-2 7B model using standard AR achieved an average Exact Match (EM) score of only 14.9%. Implementing D-AR more than doubled this performance to 30.4%.
- Scaling Mitigates Bias: Larger models are more robust. For Llama-2 70B, D-AR nearly eliminated the positional bias, with the performance gap between the first and last sentences of a document shrinking to under 2%.
- Efficiency and Architecture: Mistral 7B trained with D-AR (44.5% EM) outperformed the much larger Llama-2 70B trained with standard AR (31.7% EM). Furthermore, Mistral and Zephyr architectures were found to be inherently more robust to positional bias than the Llama-2 and Llama-3 families.
- Attention Sharpening: Analysis of attention head entropy revealed that D-AR forces the model to focus more sharply on relevant tokens, rather than diffusing attention across a long, potentially irrelevant prefix.
Discussion and Future Work
While D-AR significantly improves knowledge extraction, the study highlights that the "curse" intensifies with document length. Current results focus on fine-tuning; while the authors hypothesize that D-AR would benefit pre-training from scratch, they note the risk of under-fitting if applied too aggressively to "clean" sequences. Future research will explore the scalability of these techniques to book-length training data and their integration into large-scale pre-training regimes.
Summary updated