Training with "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-Context Tasks

Yijiong Yu, Yongfeng Huang, Zhixiao Qi, Zhe ZhouAAAI 2025 · OpenReview

Abstract

Large Language Models (LLMs) frequently struggle with the "lost in the middle" phenomenon, where information located in the center of a long context is ignored or poorly utilized. This paper proposes a novel fine-tuning strategy termed Original Text Paraphrasing (OTP). By training models to explicitly retrieve and restate relevant source text before generating a final answer, OTP significantly enhances performance in long-context tasks—achieving gains of up to 8.48%—without compromising general reasoning capabilities.

Introduction

As context windows for LLMs expand to 32k tokens and beyond, a performance paradox has emerged. While models can often pass simple "Needle in a Haystack" tests, they frequently fail at complex multi-document Question Answering (QA). The authors argue that this "lost in the middle" issue is not a lack of raw retrieval capacity, but a failure to activate that capability during complex reasoning. When retrieval remains implicit, models may fail to attend to the correct context segment in the initial steps of processing. OTP seeks to make this retrieval step explicit and reliable.

Methods: Original Text Paraphrasing

The core innovation of this work is the Original Text Paraphrasing (OTP) fine-tuning approach. Unlike standard fine-tuning, which maps a question directly to a concise answer, OTP structures the model's response into a three-part evidentiary chain:

  1. Document Indexing: The model identifies the specific document containing the answer (e.g., "According to Document [20]...").
  2. Paraphrasing: The model restates or paraphrases the relevant sentences from the source text.
  3. Final Answer: The model provides the final answer based on the retrieved and restated evidence.
Comparison between standard training flow and the proposed OTP method.

By forcing the model to "show its work" regarding retrieval, the training process prioritizes the extraction of key information before synthesis. The researchers utilized GPT-4 to generate a high-quality synthetic dataset of 5,000 QA pairs and fine-tuned models such as Llama-3 and Qwen using QLoRA.

Examples of different response formats, highlighting the detailed evidence provided by Original Text Paraphrasing.

Experimental Setup and Datasets

The study employed a bilingual (English and Chinese) training dataset consisting of 2,000 short-form instructions and 3,000 long-context samples derived from CommonCrawl and Wudao. Evaluation was conducted across several benchmarks:

  • LongBench: 19 tasks covering multi-doc QA, summarization, and few-shot learning.
  • NaturalQuestions Multi-document-QA: Specifically used to test accuracy based on the "gold" document's position within a 20-document stack.
  • MMLU: To ensure general multitask language understanding was preserved.

Key Results and Insights

The OTP method yielded substantial improvements across all tested benchmarks:

  • Performance Gains: On LongBench, the fine-tuned Qwen1.5-4B model saw an average score increase of 8.48%, while Llama3-8B improved by 5.76%.
  • Mitigating "Lost in the Middle": In NaturalQuestions testing, baseline models showed a sharp performance drop when the key document was placed in the middle (positions 5–15). OTP-tuned models maintained significantly higher and more consistent accuracy across all document positions.
  • Minimal Skill Degradation: The models showed a maximum decline of less than 2% on MMLU, suggesting that OTP is a safe optimization that does not cause catastrophic forgetting of general knowledge.

Discussion and Limitations

While effective, the OTP method introduces certain trade-offs. The requirement to paraphrase source text increases output length, leading to higher inference latency (approximately 1-3 seconds per request) and increased token costs. Additionally, while the method excels at "short-dependency" tasks (where the answer is in one specific document), further research is needed to determine its efficacy in tasks requiring the synthesis of information scattered across multiple disparate documents.

Conclusion

Original Text Paraphrasing provides a lightweight and effective mechanism for improving LLM retrieval in long-context scenarios. By transforming implicit retrieval into an explicit, structured step, OTP ensures that models leverage their full context window effectively, bridging the gap between raw capacity and task performance.

Summary updated

BibTeX
@inproceedings{yu2025training,
  author = {Yijiong Yu and Yongfeng Huang and Zhixiao Qi and Zhe Zhou},
  title = {{Training with "Paraphrasing the Original Text" Teaches LLM to Better Retrieve in Long-Context Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=y01bW5n1UM},
}
Figure