Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model

Kuniaki Saito, Chen-Yu Lee, Kihyuk Sohn, Yoshitaka UshikuNAACL 2025 · OpenReview

Abstract

Language models (LMs) store diverse factual knowledge in their parameters during self-supervised training on unlabeled documents. While instruction-tuning makes this knowledge extractable, LMs often suffer from the "perplexity curse": despite minimizing document perplexity during training, they struggle to extract information via question prompts. This study investigates this phenomenon by fine-tuning LMs on new data, revealing a significant positional bias. Models struggle to answer questions about information located in the middle or at the end of training documents. We demonstrate that this issue stems from standard auto-regressive training and propose that adding regularization, specifically Denoising Auto-Regressive (D-AR) training, mitigates the bias. Our findings provide a key to more effectively extracting parametric knowledge from LMs.

The Perplexity Curse and Positional Bias

A common assumption in LLM training is that low perplexity—the ability to predict the next token with high confidence—equates to successful knowledge acquisition. However, researchers have identified a "perplexity curse" where a model can recite a training document verbatim but fail to answer simple questions about facts contained within it.

Figure 1: Illustration of the 'perplexity curse' and positional bias in knowledge extraction.

This failure is deeply tied to the position of the information. As shown in Figure 1, if a fact is moved from the beginning of a document to the end during training, the model's ability to retrieve that specific knowledge at inference time drops significantly. This suggests that auto-regressive models become overly dependent on specific token prefixes. When an inference prompt (like a question) does not match the specific prefix seen during training, the model fails to retrieve the "parametric knowledge" stored in its weights.

Regularization Strategies for Knowledge Extraction

To break the rigid dependency on specific prefixes and improve knowledge extraction across all document positions, the authors investigated four training strategies:

Figure 2: Comparison of standard auto-regressive training versus proposed regularization techniques.
  1. Standard Auto-Regressive (AR): The baseline method of next-token prediction.
  2. Denoising Auto-Regressive (D-AR): The primary proposal. During training, $R\%$ of input tokens are replaced with random tokens. Crucially, the model must still predict the original next token. This forces the model to rely on broader factual connections rather than specific local sequences.
  3. Sentence Shuffling: Randomizing the order of sentences within a document to prevent the model from learning a fixed global position for facts.
  4. Attention Dropout: Randomly dropping elements of the attention mask to reduce the model's reliance on specific preceding tokens.

Experimental Setup and Datasets

The study utilized three distinct datasets to isolate positional effects:

  • SynthLang: A synthetic environment where models learn five languages spoken by different individuals, allowing for controlled testing of recall based on the order of mention.
  • Wiki2023+: Wikipedia articles published in 2023 (post-dating the models' original pre-training) with annotated QA pairs linked to specific sentences.
  • SynthID: A high-difficulty task requiring the memorization of ten unique 10-character identification numbers for different individuals.

Key Results and Analysis

The experiments yielded several critical insights into how LLMs store and retrieve information:

  • Significant Accuracy Gains: On the Wiki2023+ dataset, a Llama-2 7B model using standard AR achieved an average Exact Match (EM) score of only 14.9%. Implementing D-AR more than doubled this performance to 30.4%.
  • Scaling Mitigates Bias: Larger models are more robust. For Llama-2 70B, D-AR nearly eliminated the positional bias, with the performance gap between the first and last sentences of a document shrinking to under 2%.
  • Efficiency and Architecture: Mistral 7B trained with D-AR (44.5% EM) outperformed the much larger Llama-2 70B trained with standard AR (31.7% EM). Furthermore, Mistral and Zephyr architectures were found to be inherently more robust to positional bias than the Llama-2 and Llama-3 families.
  • Attention Sharpening: Analysis of attention head entropy revealed that D-AR forces the model to focus more sharply on relevant tokens, rather than diffusing attention across a long, potentially irrelevant prefix.

Discussion and Future Work

While D-AR significantly improves knowledge extraction, the study highlights that the "curse" intensifies with document length. Current results focus on fine-tuning; while the authors hypothesize that D-AR would benefit pre-training from scratch, they note the risk of under-fitting if applied too aggressively to "clean" sequences. Future research will explore the scalability of these techniques to book-length training data and their integration into large-scale pre-training regimes.

Summary updated

Abstract

Language model (LM) stores diverse factual knowledge in their parameters, which is learned during self-supervised training on unlabeled documents and is made extractable by instruction-tuning. For knowledge-intensive tasks, it is essential to memorize information in a way that makes it extractable from LM’s parameters with diverse queries. However, LMs suffer from a phenomenon called “perplexity curse”; despite minimizing document perplexity during training, LMs struggle to extract information via a question prompt. In this paper, we study the problem by fine-tuning LMs for new data and find a very intriguing fact that all studied LMs suffer from positional bias in the training document, i.e., they struggle to answer questions about the information described in the middle or at the end of the training document. Our study indicates that this problem stems from the auto-regressive training, ie., predicting the next token given all previous tokens, thus adding regularization mitigates the issue. Our discoveries supported by extensive analysis will be an important key to extracting knowledge from the parameters of LMs. We will publish our code and dataset upon acceptance.
BibTeX
@inproceedings{saito2025where,
  author = {Kuniaki Saito and Chen-Yu Lee and Kihyuk Sohn and Yoshitaka Ushiku},
  title = {{Where is the answer? An empirical study of positional bias for parametric knowledge extraction in language model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=IAaZfWzMkh},
}
Figure