Abstract
Large Language Models (LLMs) frequently struggle to utilize information located in the middle of long contexts, a phenomenon known as position bias. This paper introduces Pos2Distill, a novel framework that leverages a model's superior performance at advantageous positions (the start and end of a context) to supervise and rectify its behavior at disadvantaged middle positions. By employing inter-position knowledge distillation, Pos2Distill significantly improves positional uniformity with minimal training data, effectively using the model's own strengths to mitigate its inherent weaknesses.
Introduction
Despite the expansion of context windows in modern LLMs to 128k tokens and beyond, their ability to process information remains non-uniform. Models exhibit a significant Position Bias (PB), often referred to as the "lost-in-the-middle" problem. While information at the "edges" (the beginning and the very end of the input) is retrieved and processed with high accuracy, information buried in the center is frequently overlooked.
Existing solutions typically involve architectural modifications or expensive large-scale contextual training. Pos2Distill offers a more efficient alternative by focusing on "positional uniformity"—ensuring consistent performance regardless of where the "gold" information is located within the prompt.
The Mechanics of Position Bias
The authors identify two distinct failure modes caused by position bias:
- Token-Shifting (Retrieval): In simple lookup or retrieval tasks, the model often selects the incorrect token at a critical "turning point" during generation, leading to a total failure of the retrieval process.
- Thought-Shifting (Reasoning): In complex multi-hop reasoning tasks, the bias causes the model’s Chain-of-Thought (CoT) trajectory to deviate from the correct logical path, resulting in incorrect conclusions.
The Pos2Distill Framework
The core philosophy of Pos2Distill is inspired by the proverb: "Who tied the bell could be the one to untie it." Since the model already demonstrates the ability to process information correctly when it is placed at the start of a document (the "sink" position), these successful outputs can serve as supervisory signals for the middle positions.
Pos2Distill-R¹: Addressing Retrieval
For retrieval-heavy tasks, Pos2Distill-R¹ uses Kullback-Leibler (KL) divergence to align the probability distribution of tokens at disadvantaged middle positions with the distribution produced at the optimal start position. This includes a Position-Aware Alignment strategy, which dynamically weights training based on the difficulty of fixing specific positions.
Pos2Distill-R²: Addressing Reasoning
For complex reasoning, Pos2Distill-R² distills high-quality CoT reasoning trajectories from advantageous positions. These trajectories guide and rectify the reasoning process when the model is forced to work with information located in less favorable positions.
Experimental Evaluation
The framework was evaluated across a diverse set of benchmarks, including Retrieval-Augmented QA (NaturalQuestions, TriviaQA, WebQA), Long-Context Retrieval (KV Retrieval), and Multi-hop Reasoning (MuSiQue, 2WikiMultiHopQA, HotpotQA). The researchers tested several open-source models, including Mistral-7B-Instruct-v0.3, Qwen1.5/2.5 (7B to 32B), and Llama-3-8B-Instruct.
Key Findings
- Extreme Data Efficiency: One of the most significant advantages of Pos2Distill is its efficiency. Training on as few as 250 to 500 examples was sufficient to boost performance and close the position gap.
- Performance Gains: Pos2Distill-R¹ increased Mistral-7B's performance on the NaturalQuestions dataset by 6.7% using only 250 instances. On the WebQ dataset, Llama-3-8B achieved an average accuracy of 56.7% across all 20 positions, nearly matching its peak optimal performance of 57.9%.
- Scalability and Generalization: The method scaled effectively to 32B parameter models, reducing the retrieval performance gap from 7.2% to 2.6%. Furthermore, models trained on 20-document contexts maintained high accuracy when generalized to 50 and 80 documents.
- Cross-Task Synergy: Optimizing for retrieval (R¹) also improved reasoning performance, and vice versa, suggesting that mitigating position bias enhances the model's fundamental contextual fidelity.
Conclusion
Pos2Distill demonstrates that the "lost-in-the-middle" problem can be addressed without massive datasets or architectural overhauls. By using the model's own successful edge-case behaviors as a teacher, researchers can achieve positional uniformity efficiently. Future work may involve more granular calibration for reasoning chains and exploring adaptive strategies for black-box models where fine-tuning is not an option.
Summary updated