Abstract
Large Language Models (LLMs) frequently struggle with the "lost in the middle" phenomenon, where information located in the center of a long context is ignored or poorly utilized. This paper proposes a novel fine-tuning strategy termed Original Text Paraphrasing (OTP). By training models to explicitly retrieve and restate relevant source text before generating a final answer, OTP significantly enhances performance in long-context tasks—achieving gains of up to 8.48%—without compromising general reasoning capabilities.
Introduction
As context windows for LLMs expand to 32k tokens and beyond, a performance paradox has emerged. While models can often pass simple "Needle in a Haystack" tests, they frequently fail at complex multi-document Question Answering (QA). The authors argue that this "lost in the middle" issue is not a lack of raw retrieval capacity, but a failure to activate that capability during complex reasoning. When retrieval remains implicit, models may fail to attend to the correct context segment in the initial steps of processing. OTP seeks to make this retrieval step explicit and reliable.
Methods: Original Text Paraphrasing
The core innovation of this work is the Original Text Paraphrasing (OTP) fine-tuning approach. Unlike standard fine-tuning, which maps a question directly to a concise answer, OTP structures the model's response into a three-part evidentiary chain:
- Document Indexing: The model identifies the specific document containing the answer (e.g., "According to Document [20]...").
- Paraphrasing: The model restates or paraphrases the relevant sentences from the source text.
- Final Answer: The model provides the final answer based on the retrieved and restated evidence.
By forcing the model to "show its work" regarding retrieval, the training process prioritizes the extraction of key information before synthesis. The researchers utilized GPT-4 to generate a high-quality synthetic dataset of 5,000 QA pairs and fine-tuned models such as Llama-3 and Qwen using QLoRA.
Experimental Setup and Datasets
The study employed a bilingual (English and Chinese) training dataset consisting of 2,000 short-form instructions and 3,000 long-context samples derived from CommonCrawl and Wudao. Evaluation was conducted across several benchmarks:
- LongBench: 19 tasks covering multi-doc QA, summarization, and few-shot learning.
- NaturalQuestions Multi-document-QA: Specifically used to test accuracy based on the "gold" document's position within a 20-document stack.
- MMLU: To ensure general multitask language understanding was preserved.
Key Results and Insights
The OTP method yielded substantial improvements across all tested benchmarks:
- Performance Gains: On LongBench, the fine-tuned Qwen1.5-4B model saw an average score increase of 8.48%, while Llama3-8B improved by 5.76%.
- Mitigating "Lost in the Middle": In NaturalQuestions testing, baseline models showed a sharp performance drop when the key document was placed in the middle (positions 5–15). OTP-tuned models maintained significantly higher and more consistent accuracy across all document positions.
- Minimal Skill Degradation: The models showed a maximum decline of less than 2% on MMLU, suggesting that OTP is a safe optimization that does not cause catastrophic forgetting of general knowledge.
Discussion and Limitations
While effective, the OTP method introduces certain trade-offs. The requirement to paraphrase source text increases output length, leading to higher inference latency (approximately 1-3 seconds per request) and increased token costs. Additionally, while the method excels at "short-dependency" tasks (where the answer is in one specific document), further research is needed to determine its efficacy in tasks requiring the synthesis of information scattered across multiple disparate documents.
Conclusion
Original Text Paraphrasing provides a lightweight and effective mechanism for improving LLM retrieval in long-context scenarios. By transforming implicit retrieval into an explicit, structured step, OTP ensures that models leverage their full context window effectively, bridging the gap between raw capacity and task performance.
Summary updated